2024-09-15T02:53:31.861 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T02:53:31.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T02:53:31.970 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} email: skanta@redhat.com first_in_suite: false job_id: '7905213' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T02:53:31.970 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T02:53:31.972 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T02:53:31.972 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T02:53:31.974 INFO:teuthology.task.internal:Checking packages... 2024-09-15T02:53:31.996 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T02:53:31.996 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T02:53:31.996 INFO:teuthology.packaging:ref: None 2024-09-15T02:53:31.996 INFO:teuthology.packaging:tag: None 2024-09-15T02:53:31.996 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:53:31.996 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:53:31.997 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:53:32.147 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:53:32.148 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T02:53:32.153 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T02:53:32.153 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T02:53:32.161 INFO:teuthology.task.internal:Saving configuration 2024-09-15T02:53:32.176 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T02:53:32.187 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T02:53:32.212 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:47:56.571585', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-15T02:53:32.212 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T02:53:32.220 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-15T02:53:32.220 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T02:53:32.256 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc79d10fac0>, signals=[15]) 2024-09-15T02:53:32.256 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T02:53:32.265 INFO:teuthology.task.internal:Opening connections... 2024-09-15T02:53:32.265 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T02:53:32.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:53:32.372 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T02:53:32.380 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-09-15T02:53:32.503 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-09-15T02:53:32.503 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="20.04" 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T02:53:32.513 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T02:53:32.514 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T02:53:32.514 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=focal 2024-09-15T02:53:32.514 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=focal 2024-09-15T02:53:32.514 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-09-15T02:53:32.535 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T02:53:32.544 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T02:53:32.550 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T02:53:32.550 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:53:32.561 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T02:53:32.567 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T02:53:32.567 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:53:32.586 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T02:53:32.661 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T02:53:32.674 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T02:53:32.674 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T02:53:32.674 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:53:32.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:53:32.680 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-09-15T02:53:32.687 INFO:teuthology.orchestra.run.smithi060.stdout:5.4.0-124-generic 2024-09-15T02:53:32.687 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.4.0-124-generic 2024-09-15T02:53:32.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2024-09-15T02:53:32.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2024-09-15T02:53:33.074 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:53:33.082 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-15T02:53:33.133 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-15T02:53:33.485 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-15T02:53:33.591 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-15T02:53:33.741 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-15T02:53:33.949 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-15T02:53:34.068 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-15T02:53:34.141 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-15T02:53:34.155 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-15T02:53:34.156 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-15T02:53:34.247 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-15T02:53:34.247 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-15T02:53:34.248 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-15T02:53:34.261 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-15T02:53:34.262 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:53:34.262 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-15T02:53:34.262 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-15T02:53:34.288 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-15T02:53:34.306 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-15T02:53:34.313 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-15T02:53:34.318 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-15T02:53:34.318 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-15T02:53:34.357 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-15T02:53:34.357 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-15T02:53:34.365 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:53:34.365 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-15T02:53:34.386 INFO:teuthology.orchestra.run.smithi060.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-15T02:53:34.387 INFO:teuthology.orchestra.run.smithi060.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-15T02:53:34.398 INFO:teuthology.orchestra.run.smithi060.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-15T02:53:34.428 INFO:teuthology.orchestra.run.smithi060.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-15T02:53:34.434 INFO:teuthology.orchestra.run.smithi060.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-15T02:53:34.435 INFO:teuthology.orchestra.run.smithi060.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-15T02:53:34.435 INFO:teuthology.orchestra.run.smithi060.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-15T02:53:34.495 INFO:teuthology.orchestra.run.smithi060.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-15T02:53:34.496 INFO:teuthology.orchestra.run.smithi060.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-15T02:53:34.496 INFO:teuthology.orchestra.run.smithi060.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-15T02:53:34.497 INFO:teuthology.orchestra.run.smithi060.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-15T02:53:34.498 INFO:teuthology.orchestra.run.smithi060.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-15T02:53:34.577 INFO:teuthology.orchestra.run.smithi060.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-15T02:53:34.577 INFO:teuthology.orchestra.run.smithi060.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-15T02:53:34.578 INFO:teuthology.orchestra.run.smithi060.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-15T02:53:38.638 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 21.8 MB in 4s (5,928 kB/s) 2024-09-15T02:53:40.399 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-15T02:53:40.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:53:40.515 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-15T02:53:40.726 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2024-09-15T02:53:40.727 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2024-09-15T02:53:40.945 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2024-09-15T02:53:40.945 INFO:teuthology.orchestra.run.smithi060.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:53:40.945 INFO:teuthology.orchestra.run.smithi060.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-15T02:53:40.945 INFO:teuthology.orchestra.run.smithi060.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-15T02:53:40.946 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2024-09-15T02:53:40.946 INFO:teuthology.orchestra.run.smithi060.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-15T02:53:40.946 INFO:teuthology.orchestra.run.smithi060.stdout:Recommended packages: 2024-09-15T02:53:40.946 INFO:teuthology.orchestra.run.smithi060.stdout: thermald 2024-09-15T02:53:41.013 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2024-09-15T02:53:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:53:41.013 INFO:teuthology.orchestra.run.smithi060.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-15T02:53:41.014 INFO:teuthology.orchestra.run.smithi060.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-15T02:53:41.015 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2024-09-15T02:53:41.016 INFO:teuthology.orchestra.run.smithi060.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T02:53:41.070 INFO:teuthology.orchestra.run.smithi060.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:53:41.070 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 77.2 MB of archives. 2024-09-15T02:53:41.070 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-15T02:53:41.071 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-15T02:53:41.400 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-15T02:53:41.494 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-15T02:53:41.960 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-15T02:53:41.960 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-15T02:53:41.960 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-15T02:53:42.055 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-15T02:53:42.065 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-15T02:53:42.853 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 77.2 MB in 1s (74.0 MB/s) 2024-09-15T02:53:43.058 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-15T02:53:44.772 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-15T02:53:44.777 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:53:44.938 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:53:46.914 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-15T02:53:46.927 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:53:47.010 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:53:48.117 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-15T02:53:48.130 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:53:48.158 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:53:53.088 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:53:53.240 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:53:53.634 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:53:53.760 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:53:54.138 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-15T02:53:54.154 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-15T02:53:54.204 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:54:00.081 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-15T02:54:00.102 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:54:00.148 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:54:04.223 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:54:04.353 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:54:04.739 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:54:06.009 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:54:07.189 INFO:teuthology.orchestra.run.smithi060.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:54:07.189 INFO:teuthology.orchestra.run.smithi060.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-15T02:54:07.345 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:54:09.636 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:54:09.765 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-15T02:54:09.916 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:54:10.042 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-15T02:54:10.188 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-15T02:54:10.328 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:54:10.397 INFO:teuthology.orchestra.run.smithi060.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T02:54:10.397 INFO:teuthology.orchestra.run.smithi060.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:54:33.266 INFO:teuthology.orchestra.run.smithi060.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T02:54:33.267 INFO:teuthology.orchestra.run.smithi060.stdout:Sourcing file `/etc/default/grub' 2024-09-15T02:54:33.291 INFO:teuthology.orchestra.run.smithi060.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:54:33.315 INFO:teuthology.orchestra.run.smithi060.stdout:Generating grub configuration file ... 2024-09-15T02:54:33.704 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:54:33.726 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:54:34.119 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:54:34.130 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:54:36.677 INFO:teuthology.orchestra.run.smithi060.stdout:done 2024-09-15T02:54:37.150 DEBUG:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 20 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 5.4.0.195.193 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:54:37.191 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:54:37.192 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2024-09-15T02:54:37.192 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2024-09-15T02:54:37.192 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:54:37.192 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2024-09-15T02:54:37.192 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-15T02:54:37.192 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:54:37.192 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:54:37.192 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T02:54:37.193 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:54:37.193 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2024-09-15T02:54:37.280 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2024-09-15T02:54:37.410 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:54:37.417 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:54:37.435 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:54:37.547 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:54:39.318 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-15T02:54:39.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:54:39.417 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-15T02:54:39.642 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2024-09-15T02:54:39.643 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2024-09-15T02:54:39.899 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-15T02:54:39.899 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:54:39.901 DEBUG:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2024-09-15T02:54:39.922 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2024-09-15T02:54:39.922 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 20 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 5.4.0.195.193 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2024-09-15T02:54:39.923 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:54:39.924 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2024-09-15T02:54:39.924 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-15T02:54:39.929 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.cf85oB7M4C 2024-09-15T02:54:39.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cf85oB7M4C 2024-09-15T02:54:39.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.cf85oB7M4C 2024-09-15T02:54:40.031 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.cf85oB7M4C is 9KB 2024-09-15T02:54:40.083 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.cf85oB7M4C 2024-09-15T02:54:40.092 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T02:54:40.108 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T02:54:40.108 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T02:54:40.108 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T02:54:40.187 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-15T02:54:40.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo update-grub 2024-09-15T02:54:41.652 INFO:teuthology.orchestra.run.smithi060.stderr:Sourcing file `/etc/default/grub' 2024-09-15T02:54:41.655 INFO:teuthology.orchestra.run.smithi060.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:54:41.672 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-09-15T02:54:41.942 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:54:41.954 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:54:42.336 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:54:42.348 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:54:43.785 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-09-15T02:54:43.787 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-09-15T02:55:13.817 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T02:55:13.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-15T02:55:13.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:56:13.878 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T02:56:22.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-15T02:56:22.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:56:23.325 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-15T02:56:24.499 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-15T02:56:24.499 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-15T02:56:24.500 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-09-15T02:56:24.552 INFO:teuthology.orchestra.run.smithi060.stdout:5.4.0-195-generic 2024-09-15T02:56:24.552 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-15T02:56:24.552 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:56:24.552 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T02:56:25.553 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T02:56:25.553 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:56:25.853 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-09-15T02:56:25.881 DEBUG:teuthology.parallel:result is None 2024-09-15T02:56:25.881 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T02:56:25.888 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T02:56:25.889 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:56:25.896 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T02:56:25.902 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T02:56:25.910 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T02:56:25.910 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:56:25.967 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T02:56:25.976 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T02:56:25.976 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:56:26.002 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:26.013 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:56:26.014 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T02:56:26.021 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T02:56:26.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:56:26.072 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T02:56:26.156 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T02:56:26.156 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:56:26.163 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:56:26.170 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:56:26.224 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T02:56:26.225 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:56:26.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-09-15T02:56:26.294 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T02:56:26.301 INFO:teuthology.task.internal:Starting timer... 2024-09-15T02:56:26.301 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T02:56:26.313 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T02:56:26.324 DEBUG:teuthology.task.selinux:Excluding smithi060: OS 'ubuntu' does not support SELinux 2024-09-15T02:56:26.325 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T02:56:26.325 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-15T02:56:26.325 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T02:56:26.325 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T02:56:26.334 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T02:56:26.346 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T02:56:26.347 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T03:02:12.571 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com')] 2024-09-15T03:02:12.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-15T03:02:12.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:02:12.674 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-15T03:02:12.798 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-15T03:02:12.799 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T03:02:12.810 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T03:02:12.811 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:02:12.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:02:12.847 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-15T03:02:12.847 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: Command line: ntpd -gq 2024-09-15T03:02:12.848 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: proto: precision = 0.076 usec (-24) 2024-09-15T03:02:12.848 INFO:teuthology.orchestra.run.smithi060.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T03:02:12.848 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T03:02:12.849 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T03:02:12.849 INFO:teuthology.orchestra.run.smithi060.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T03:02:12.850 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T03:02:12.850 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T03:02:12.850 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T03:02:12.850 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: Listen normally on 3 ens1f0 172.21.15.60:123 2024-09-15T03:02:12.850 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: Listen normally on 4 lo [::1]:123 2024-09-15T03:02:12.850 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1560%4]:123 2024-09-15T03:02:12.850 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:12 ntpd[13627]: Listening on routing socket on fd #22 for interface updates 2024-09-15T03:02:14.848 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:14 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:14.848 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:14 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:15.848 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:15 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:15.848 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:15 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:15.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:15 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:15.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:15 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:15.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:15 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:15.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:15 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:17.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:17.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:17.851 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:17.851 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:17.851 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:17.851 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:17.851 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:17 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:19.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:19.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:19.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:19.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:19.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:19.851 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:19 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:21.849 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:21.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:21.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T03:02:21.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T03:02:21.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T03:02:21.850 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T03:02:21.850 INFO:teuthology.orchestra.run.smithi060.stdout:15 Sep 03:02:21 ntpd[13627]: ntpd: time slew +0.002600 s 2024-09-15T03:02:21.850 INFO:teuthology.orchestra.run.smithi060.stdout:ntpd: time slew +0.002600s 2024-09-15T03:02:21.851 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T03:02:21.851 INFO:teuthology.orchestra.run.smithi060.stderr:15 Sep 03:02:21 ntpd[13627]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T03:02:21.957 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T03:02:21.958 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2024-09-15T03:02:21.958 INFO:teuthology.orchestra.run.smithi060.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T03:02:21.958 INFO:teuthology.orchestra.run.smithi060.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T03:02:21.958 INFO:teuthology.orchestra.run.smithi060.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T03:02:21.958 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T03:02:21.959 INFO:teuthology.run_tasks:Running task install... 2024-09-15T03:02:21.967 DEBUG:teuthology.task.install:project ceph 2024-09-15T03:02:21.967 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:02:21.967 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T03:02:21.967 INFO:teuthology.task.install:Using flavor: default 2024-09-15T03:02:21.973 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T03:02:21.973 INFO:teuthology.task.install:extra packages: [] 2024-09-15T03:02:21.974 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-key list | grep Ceph 2024-09-15T03:02:22.102 INFO:teuthology.orchestra.run.smithi060.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-15T03:02:22.167 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T03:02:22.167 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T03:02:22.168 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T03:02:22.168 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:02:22.320 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-15T03:02:22.320 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-15T03:02:22.431 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:02:22.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T03:02:22.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo apt-get update 2024-09-15T03:02:22.640 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T03:02:22.694 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-15T03:02:22.715 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-15T03:02:22.739 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-15T03:02:22.756 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T03:02:22.761 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-15T03:02:22.794 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-15T03:02:22.837 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T03:02:22.919 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T03:02:23.325 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 28.5 kB in 1s (43.1 kB/s) 2024-09-15T03:02:24.518 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-15T03:02:24.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-15T03:02:24.619 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2024-09-15T03:02:24.840 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2024-09-15T03:02:24.841 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2024-09-15T03:02:25.042 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T03:02:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-15T03:02:25.044 INFO:teuthology.orchestra.run.smithi060.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T03:02:25.044 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2024-09-15T03:02:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T03:02:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-15T03:02:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-15T03:02:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-15T03:02:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-15T03:02:25.045 INFO:teuthology.orchestra.run.smithi060.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: socat xmlstarlet 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2024-09-15T03:02:25.046 INFO:teuthology.orchestra.run.smithi060.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout:Recommended packages: 2024-09-15T03:02:25.047 INFO:teuthology.orchestra.run.smithi060.stdout: btrfs-tools 2024-09-15T03:02:25.170 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2024-09-15T03:02:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T03:02:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T03:02:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-15T03:02:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-15T03:02:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-15T03:02:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-15T03:02:25.170 INFO:teuthology.orchestra.run.smithi060.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-15T03:02:25.171 INFO:teuthology.orchestra.run.smithi060.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-15T03:02:25.172 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-15T03:02:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-15T03:02:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-15T03:02:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-15T03:02:25.173 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2024-09-15T03:02:25.173 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 librbd1 2024-09-15T03:02:25.239 INFO:teuthology.orchestra.run.smithi060.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-15T03:02:25.239 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 201 MB of archives. 2024-09-15T03:02:25.239 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-15T03:02:25.239 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-15T03:02:25.322 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-15T03:02:25.323 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-15T03:02:25.324 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-15T03:02:25.332 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-15T03:02:25.335 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-15T03:02:25.363 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-15T03:02:25.364 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-15T03:02:25.369 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-15T03:02:25.401 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-15T03:02:25.403 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-15T03:02:25.409 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-15T03:02:25.411 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-15T03:02:25.411 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-15T03:02:25.414 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-15T03:02:25.414 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-15T03:02:25.414 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-15T03:02:25.415 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-15T03:02:25.419 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-15T03:02:25.434 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-15T03:02:25.434 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-15T03:02:25.434 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-15T03:02:25.435 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-15T03:02:25.436 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T03:02:25.437 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-15T03:02:25.437 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-15T03:02:25.439 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-15T03:02:25.439 INFO:teuthology.orchestra.run.smithi060.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-15T03:02:25.453 INFO:teuthology.orchestra.run.smithi060.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-15T03:02:25.454 INFO:teuthology.orchestra.run.smithi060.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-15T03:02:25.454 INFO:teuthology.orchestra.run.smithi060.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-15T03:02:25.470 INFO:teuthology.orchestra.run.smithi060.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-15T03:02:25.470 INFO:teuthology.orchestra.run.smithi060.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-15T03:02:25.471 INFO:teuthology.orchestra.run.smithi060.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-15T03:02:25.472 INFO:teuthology.orchestra.run.smithi060.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-15T03:02:25.473 INFO:teuthology.orchestra.run.smithi060.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-15T03:02:25.477 INFO:teuthology.orchestra.run.smithi060.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-15T03:02:25.479 INFO:teuthology.orchestra.run.smithi060.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T03:02:25.487 INFO:teuthology.orchestra.run.smithi060.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-15T03:02:25.488 INFO:teuthology.orchestra.run.smithi060.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-15T03:02:25.489 INFO:teuthology.orchestra.run.smithi060.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-15T03:02:25.494 INFO:teuthology.orchestra.run.smithi060.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-15T03:02:25.504 INFO:teuthology.orchestra.run.smithi060.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-15T03:02:25.505 INFO:teuthology.orchestra.run.smithi060.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-15T03:02:25.505 INFO:teuthology.orchestra.run.smithi060.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-15T03:02:25.544 INFO:teuthology.orchestra.run.smithi060.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-15T03:02:25.557 INFO:teuthology.orchestra.run.smithi060.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-15T03:02:25.564 INFO:teuthology.orchestra.run.smithi060.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-15T03:02:25.564 INFO:teuthology.orchestra.run.smithi060.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-15T03:02:25.567 INFO:teuthology.orchestra.run.smithi060.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-15T03:02:25.568 INFO:teuthology.orchestra.run.smithi060.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-15T03:02:25.573 INFO:teuthology.orchestra.run.smithi060.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-15T03:02:25.583 INFO:teuthology.orchestra.run.smithi060.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-15T03:02:25.598 INFO:teuthology.orchestra.run.smithi060.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-15T03:02:25.600 INFO:teuthology.orchestra.run.smithi060.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-15T03:02:25.611 INFO:teuthology.orchestra.run.smithi060.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-15T03:02:25.697 INFO:teuthology.orchestra.run.smithi060.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-15T03:02:25.697 INFO:teuthology.orchestra.run.smithi060.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-15T03:02:25.698 INFO:teuthology.orchestra.run.smithi060.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-15T03:02:25.700 INFO:teuthology.orchestra.run.smithi060.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-15T03:02:25.700 INFO:teuthology.orchestra.run.smithi060.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-15T03:02:25.700 INFO:teuthology.orchestra.run.smithi060.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-15T03:02:25.701 INFO:teuthology.orchestra.run.smithi060.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-15T03:02:25.702 INFO:teuthology.orchestra.run.smithi060.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-15T03:02:25.704 INFO:teuthology.orchestra.run.smithi060.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-15T03:02:25.704 INFO:teuthology.orchestra.run.smithi060.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-15T03:02:25.710 INFO:teuthology.orchestra.run.smithi060.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-15T03:02:25.714 INFO:teuthology.orchestra.run.smithi060.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-15T03:02:25.723 INFO:teuthology.orchestra.run.smithi060.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-15T03:02:25.725 INFO:teuthology.orchestra.run.smithi060.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-15T03:02:25.727 INFO:teuthology.orchestra.run.smithi060.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-15T03:02:25.728 INFO:teuthology.orchestra.run.smithi060.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-15T03:02:25.731 INFO:teuthology.orchestra.run.smithi060.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-15T03:02:25.734 INFO:teuthology.orchestra.run.smithi060.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-15T03:02:25.749 INFO:teuthology.orchestra.run.smithi060.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-15T03:02:25.751 INFO:teuthology.orchestra.run.smithi060.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-15T03:02:25.788 INFO:teuthology.orchestra.run.smithi060.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-15T03:02:25.789 INFO:teuthology.orchestra.run.smithi060.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-15T03:02:25.789 INFO:teuthology.orchestra.run.smithi060.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-15T03:02:25.789 INFO:teuthology.orchestra.run.smithi060.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-15T03:02:25.790 INFO:teuthology.orchestra.run.smithi060.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-15T03:02:25.791 INFO:teuthology.orchestra.run.smithi060.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-15T03:02:25.803 INFO:teuthology.orchestra.run.smithi060.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T03:02:25.803 INFO:teuthology.orchestra.run.smithi060.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-15T03:02:25.804 INFO:teuthology.orchestra.run.smithi060.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-15T03:02:25.806 INFO:teuthology.orchestra.run.smithi060.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-15T03:02:25.823 INFO:teuthology.orchestra.run.smithi060.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-15T03:02:25.823 INFO:teuthology.orchestra.run.smithi060.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-15T03:02:25.824 INFO:teuthology.orchestra.run.smithi060.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-15T03:02:25.826 INFO:teuthology.orchestra.run.smithi060.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-15T03:02:25.826 INFO:teuthology.orchestra.run.smithi060.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-15T03:02:26.208 INFO:teuthology.orchestra.run.smithi060.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-15T03:02:26.254 INFO:teuthology.orchestra.run.smithi060.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-15T03:02:26.256 INFO:teuthology.orchestra.run.smithi060.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-15T03:02:26.257 INFO:teuthology.orchestra.run.smithi060.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-15T03:02:26.280 INFO:teuthology.orchestra.run.smithi060.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-15T03:02:26.359 INFO:teuthology.orchestra.run.smithi060.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-15T03:02:26.869 INFO:teuthology.orchestra.run.smithi060.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-15T03:02:26.870 INFO:teuthology.orchestra.run.smithi060.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-15T03:02:26.878 INFO:teuthology.orchestra.run.smithi060.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-15T03:02:26.898 INFO:teuthology.orchestra.run.smithi060.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-15T03:02:26.898 INFO:teuthology.orchestra.run.smithi060.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-15T03:02:26.899 INFO:teuthology.orchestra.run.smithi060.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-15T03:02:27.008 INFO:teuthology.orchestra.run.smithi060.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-15T03:02:27.172 INFO:teuthology.orchestra.run.smithi060.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-15T03:02:27.172 INFO:teuthology.orchestra.run.smithi060.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-15T03:02:28.521 INFO:teuthology.orchestra.run.smithi060.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-15T03:02:28.579 INFO:teuthology.orchestra.run.smithi060.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-15T03:02:28.580 INFO:teuthology.orchestra.run.smithi060.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-15T03:02:29.104 INFO:teuthology.orchestra.run.smithi060.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-15T03:02:29.772 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 201 MB in 4s (51.2 MB/s) 2024-09-15T03:02:29.947 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-15T03:02:30.004 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-15T03:02:30.009 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-15T03:02:30.088 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T03:02:30.441 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:30.550 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T03:02:31.271 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:31.429 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T03:02:32.144 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T03:02:32.158 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:32.199 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:32.702 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rados. 2024-09-15T03:02:32.719 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:32.760 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:33.166 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T03:02:33.186 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T03:02:33.230 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:33.457 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T03:02:33.477 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:33.540 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:33.893 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T03:02:33.913 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T03:02:33.951 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:34.304 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T03:02:34.321 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-15T03:02:34.363 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-15T03:02:34.640 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T03:02:34.657 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:34.698 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:35.018 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-15T03:02:35.035 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-15T03:02:35.068 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T03:02:35.446 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-15T03:02:35.463 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-15T03:02:35.504 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T03:02:35.890 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T03:02:35.907 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-15T03:02:35.948 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T03:02:36.293 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T03:02:36.311 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-15T03:02:36.368 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T03:02:36.696 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T03:02:36.715 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-15T03:02:36.754 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T03:02:37.241 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T03:02:37.257 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T03:02:37.310 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:02:37.836 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T03:02:37.853 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T03:02:37.885 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:02:38.255 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T03:02:38.274 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T03:02:38.313 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:02:38.733 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-15T03:02:38.751 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-15T03:02:38.816 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T03:02:39.186 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librgw2. 2024-09-15T03:02:39.204 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:39.244 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:40.030 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T03:02:40.050 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:40.088 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:40.466 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liboath0. 2024-09-15T03:02:40.483 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-15T03:02:40.524 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-15T03:02:40.860 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T03:02:40.878 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:40.943 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:41.597 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-common. 2024-09-15T03:02:41.617 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:41.672 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:44.369 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-base. 2024-09-15T03:02:44.387 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:44.437 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:45.382 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cryptography. 2024-09-15T03:02:45.399 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-15T03:02:45.440 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T03:02:45.844 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T03:02:45.860 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-15T03:02:45.902 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-15T03:02:46.246 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-packaging. 2024-09-15T03:02:46.263 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-15T03:02:46.304 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-15T03:02:46.582 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T03:02:46.599 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T03:02:46.640 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:47.103 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T03:02:47.219 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-15T03:02:47.219 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T03:02:47.523 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T03:02:47.540 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-15T03:02:47.581 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-15T03:02:47.934 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T03:02:47.954 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-15T03:02:47.992 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-15T03:02:48.329 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jwt. 2024-09-15T03:02:48.347 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-15T03:02:48.387 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T03:02:48.724 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T03:02:48.741 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-15T03:02:48.773 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-15T03:02:49.110 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T03:02:49.130 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-15T03:02:49.201 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-15T03:02:49.537 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-mako. 2024-09-15T03:02:49.558 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-15T03:02:49.595 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T03:02:49.941 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T03:02:49.957 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T03:02:49.999 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T03:02:50.316 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T03:02:50.335 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-15T03:02:50.374 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T03:02:50.752 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webob. 2024-09-15T03:02:50.768 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-15T03:02:50.810 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T03:02:51.154 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-15T03:02:51.172 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-15T03:02:51.213 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T03:02:51.560 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bs4. 2024-09-15T03:02:51.577 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-15T03:02:51.649 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-15T03:02:52.019 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T03:02:52.038 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-15T03:02:52.080 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T03:02:52.422 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T03:02:52.438 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-15T03:02:52.480 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-15T03:02:52.817 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-paste. 2024-09-15T03:02:52.832 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-15T03:02:52.875 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T03:02:53.186 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T03:02:53.202 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-15T03:02:53.244 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T03:02:53.576 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T03:02:53.592 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-15T03:02:53.634 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-15T03:02:53.954 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T03:02:53.971 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-15T03:02:54.012 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-15T03:02:54.366 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T03:02:54.382 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-15T03:02:54.451 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T03:02:54.829 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T03:02:54.845 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-15T03:02:54.887 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T03:02:55.257 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T03:02:55.273 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:55.315 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:55.694 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T03:02:55.710 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:55.752 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:56.272 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T03:02:56.288 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:56.330 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:02:58.067 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T03:02:58.083 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:02:58.125 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:00.403 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph. 2024-09-15T03:03:00.420 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:03:00.478 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:00.822 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T03:03:00.839 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:03:00.872 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:01.325 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T03:03:01.341 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:03:01.392 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:01.970 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package cephadm. 2024-09-15T03:03:01.986 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:03:02.028 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:02.364 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T03:03:02.381 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-15T03:03:02.422 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T03:03:02.767 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T03:03:02.782 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T03:03:02.850 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:03.220 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T03:03:03.236 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T03:03:03.286 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T03:03:03.531 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjs-underscore. 2024-09-15T03:03:03.547 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-15T03:03:03.622 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T03:03:03.899 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-15T03:03:03.917 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-15T03:03:03.991 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T03:03:04.345 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-routes. 2024-09-15T03:03:04.361 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-15T03:03:04.403 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-15T03:03:04.765 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T03:03:04.781 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T03:03:04.823 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:06.153 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-decorator. 2024-09-15T03:03:06.173 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-15T03:03:06.203 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T03:03:06.497 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-15T03:03:06.515 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-15T03:03:06.555 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T03:03:06.875 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-scipy. 2024-09-15T03:03:06.892 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-15T03:03:06.925 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-15T03:03:08.292 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-15T03:03:08.309 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-15T03:03:08.350 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T03:03:08.912 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T03:03:08.928 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-15T03:03:08.986 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-15T03:03:09.382 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T03:03:09.398 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-15T03:03:09.440 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T03:03:10.223 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T03:03:10.243 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T03:03:10.281 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:11.567 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T03:03:11.584 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-15T03:03:11.626 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-15T03:03:11.937 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T03:03:11.957 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-15T03:03:11.996 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T03:03:12.374 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T03:03:12.393 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-15T03:03:12.432 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T03:03:12.777 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T03:03:12.794 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-15T03:03:12.835 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-15T03:03:13.531 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T03:03:13.548 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-15T03:03:13.589 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-15T03:03:13.934 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-blinker. 2024-09-15T03:03:13.951 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-15T03:03:13.992 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T03:03:14.320 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-15T03:03:14.337 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-15T03:03:14.378 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T03:03:14.727 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T03:03:14.747 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-15T03:03:14.785 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T03:03:15.121 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T03:03:15.138 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-15T03:03:15.204 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T03:03:15.557 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T03:03:15.577 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-15T03:03:15.615 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T03:03:16.290 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T03:03:16.311 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T03:03:16.348 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:16.659 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T03:03:16.676 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-15T03:03:16.718 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-15T03:03:17.104 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T03:03:17.121 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T03:03:17.166 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T03:03:17.452 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package jq. 2024-09-15T03:03:17.469 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T03:03:17.510 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T03:03:17.789 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package socat. 2024-09-15T03:03:17.807 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-15T03:03:17.847 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-15T03:03:18.192 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T03:03:18.211 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-15T03:03:18.250 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-15T03:03:18.632 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-test. 2024-09-15T03:03:18.649 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:03:18.690 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:23.561 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T03:03:23.580 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T03:03:23.619 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:23.957 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-15T03:03:23.975 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T03:03:24.015 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T03:03:24.535 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T03:03:24.552 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:03:24.593 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:25.021 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T03:03:25.040 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-15T03:03:25.097 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T03:03:25.434 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-15T03:03:25.454 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T03:03:25.526 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T03:03:27.244 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-15T03:03:27.266 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-15T03:03:27.319 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-15T03:03:27.658 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-15T03:03:27.675 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-15T03:03:27.716 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T03:03:28.043 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-attr. 2024-09-15T03:03:28.062 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-15T03:03:28.101 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-15T03:03:28.438 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webencodings. 2024-09-15T03:03:28.459 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-15T03:03:28.496 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T03:03:28.807 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-html5lib. 2024-09-15T03:03:28.828 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-15T03:03:28.865 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-15T03:03:29.218 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-15T03:03:29.239 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-15T03:03:29.276 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T03:03:29.779 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T03:03:29.799 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T03:03:29.837 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T03:03:30.200 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T03:03:30.218 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-15T03:03:30.258 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-15T03:03:30.594 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T03:03:30.613 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-15T03:03:30.669 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T03:03:31.038 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-py. 2024-09-15T03:03:31.059 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-15T03:03:31.097 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T03:03:31.450 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T03:03:31.468 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-15T03:03:31.508 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T03:03:31.839 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T03:03:31.857 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-15T03:03:31.922 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T03:03:32.259 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T03:03:32.279 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-15T03:03:32.316 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-15T03:03:32.703 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T03:03:32.721 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-15T03:03:32.761 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T03:03:33.064 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T03:03:33.082 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-15T03:03:33.122 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T03:03:33.868 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package radosgw. 2024-09-15T03:03:33.888 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:03:33.927 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:35.690 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T03:03:35.710 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:03:35.756 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:36.058 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-15T03:03:36.395 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T03:03:36.554 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T03:03:36.680 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T03:03:36.917 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:37.653 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user cephadm....done 2024-09-15T03:03:37.815 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T03:03:37.966 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T03:03:38.317 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T03:03:38.602 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T03:03:38.770 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-15T03:03:38.896 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T03:03:39.272 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-15T03:03:39.741 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T03:03:40.059 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-15T03:03:40.360 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T03:03:40.686 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T03:03:41.021 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:41.355 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T03:03:41.498 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-15T03:03:41.775 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T03:03:42.101 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T03:03:42.260 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T03:03:42.636 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-15T03:03:42.929 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T03:03:43.255 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T03:03:43.623 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T03:03:43.908 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T03:03:44.484 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-15T03:03:44.777 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-15T03:03:45.086 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T03:03:45.212 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T03:03:45.363 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T03:03:45.673 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T03:03:46.100 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T03:03:46.468 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T03:03:46.845 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-15T03:03:46.971 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:47.305 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:03:47.440 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-15T03:03:47.850 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T03:03:47.976 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-15T03:03:48.085 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-15T03:03:48.395 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T03:03:48.721 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:03:48.847 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T03:03:49.023 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T03:03:49.358 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T03:03:49.659 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-15T03:03:49.968 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T03:03:50.094 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T03:03:50.387 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-15T03:03:50.730 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T03:03:51.023 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T03:03:51.374 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-15T03:03:51.634 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-15T03:03:51.785 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T03:03:52.120 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-15T03:03:52.496 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T03:03:52.597 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T03:03:52.698 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T03:03:53.066 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T03:03:53.401 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:53.560 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-15T03:03:55.066 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:55.219 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T03:03:55.351 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-15T03:03:55.736 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-15T03:03:56.029 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-15T03:03:56.314 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:03:56.415 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-15T03:03:56.758 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:56.884 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T03:03:57.244 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:57.395 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T03:03:57.721 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T03:03:58.785 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:03:58.911 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T03:03:59.037 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-15T03:03:59.180 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T03:03:59.281 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-15T03:03:59.566 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T03:03:59.700 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-15T03:04:00.043 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:00.203 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:00.428 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T03:04:00.428 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T03:04:00.896 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:01.116 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T03:04:01.426 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T03:04:01.744 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T03:04:02.105 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:02.256 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:02.365 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:02.499 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:02.659 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-15T03:04:02.977 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-15T03:04:03.270 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:03.421 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:03.547 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T03:04:04.492 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:05.120 INFO:teuthology.orchestra.run.smithi060.stdout:Adding group ceph....done 2024-09-15T03:04:05.514 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user ceph....done 2024-09-15T03:04:05.690 INFO:teuthology.orchestra.run.smithi060.stdout:Setting system user ceph properties....done 2024-09-15T03:04:05.711 INFO:teuthology.orchestra.run.smithi060.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T03:04:05.865 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T03:04:06.190 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T03:04:06.677 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:06.803 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:07.105 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:04:07.106 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:04:07.763 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:08.164 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T03:04:08.633 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:08.858 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T03:04:08.859 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T03:04:09.334 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:09.562 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T03:04:09.563 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T03:04:10.044 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:10.541 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T03:04:10.541 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T03:04:10.996 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:11.156 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:11.298 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:11.571 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T03:04:11.571 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T03:04:12.008 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:12.184 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:12.310 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:12.478 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:04:12.605 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-15T03:04:12.748 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-15T03:04:13.024 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-15T03:04:16.127 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-15T03:04:16.735 INFO:teuthology.orchestra.run.smithi060.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T03:04:16.740 DEBUG:teuthology.parallel:result is None 2024-09-15T03:04:16.741 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:04:16.895 DEBUG:teuthology.orchestra.run.smithi060:> dpkg-query -W -f '${Version}' ceph 2024-09-15T03:04:16.918 INFO:teuthology.orchestra.run.smithi060.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-15T03:04:16.919 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-15T03:04:16.919 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-15T03:04:16.921 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T03:04:16.922 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:16.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:04:16.974 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T03:04:16.975 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:16.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:04:16.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:04:17.006 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T03:04:17.007 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:17.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:04:17.065 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:04:17.075 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T03:04:17.075 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:17.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:04:17.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:04:17.098 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T03:04:17.177 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T03:04:17.177 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-09-15T03:04:17.193 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T03:04:17.193 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:04:17.204 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T03:04:17.204 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:04:17.222 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T03:04:17.222 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T03:04:17.222 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905213', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T03:04:17.223 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:04:17.264 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:04:17.281 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:17.281 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:04:17.287 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:04:17.287 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-09-15T03:04:17.336 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:04:17.336 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:04:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 358 Links: 1 2024-09-15T03:04:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:04:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-15 03:00:45.169105118 +0000 2024-09-15T03:04:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-15 03:00:44.949110179 +0000 2024-09-15T03:04:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-15 03:00:44.949110179 +0000 2024-09-15T03:04:17.337 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2024-09-15T03:04:17.337 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:04:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-15T03:04:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-15T03:04:17.349 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000310216 s, 1.7 MB/s 2024-09-15T03:04:17.350 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:04:17.358 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-09-15T03:04:17.366 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:04:17.366 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:04:17.366 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-15T03:04:17.366 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:04:17.366 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-15 03:00:45.589095455 +0000 2024-09-15T03:04:17.366 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-15 03:00:45.373100426 +0000 2024-09-15T03:04:17.366 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-15 03:00:45.373100426 +0000 2024-09-15T03:04:17.366 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2024-09-15T03:04:17.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:04:17.381 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-15T03:04:17.381 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-15T03:04:17.381 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000230938 s, 2.2 MB/s 2024-09-15T03:04:17.382 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:04:17.391 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-09-15T03:04:17.440 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:04:17.440 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:04:17.440 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 374 Links: 1 2024-09-15T03:04:17.440 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:04:17.440 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-15 03:00:46.009085792 +0000 2024-09-15T03:04:17.440 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-15 03:00:45.801090578 +0000 2024-09-15T03:04:17.440 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-15 03:00:45.801090578 +0000 2024-09-15T03:04:17.441 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2024-09-15T03:04:17.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:04:17.455 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-15T03:04:17.456 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-15T03:04:17.456 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000280074 s, 1.8 MB/s 2024-09-15T03:04:17.457 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:04:17.465 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-09-15T03:04:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:04:17.516 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:04:17.516 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 377 Links: 1 2024-09-15T03:04:17.517 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:04:17.517 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-15 03:00:46.421076313 +0000 2024-09-15T03:04:17.517 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-15 03:00:46.213081100 +0000 2024-09-15T03:04:17.517 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-15 03:00:46.213081100 +0000 2024-09-15T03:04:17.517 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2024-09-15T03:04:17.517 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:04:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-15T03:04:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-15T03:04:17.528 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000285457 s, 1.8 MB/s 2024-09-15T03:04:17.529 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:04:17.537 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:04:17.537 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-15T03:04:17.537 INFO:tasks.ceph:Generating config... 2024-09-15T03:04:17.538 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T03:04:17.539 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-15T03:04:17.539 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-15T03:04:17.539 INFO:tasks.ceph:[global] ms type = async 2024-09-15T03:04:17.539 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T03:04:17.539 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T03:04:17.539 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T03:04:17.539 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T03:04:17.539 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T03:04:17.539 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T03:04:17.539 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T03:04:17.539 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-15T03:04:17.539 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T03:04:17.539 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T03:04:17.540 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T03:04:17.541 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T03:04:17.541 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T03:04:17.541 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T03:04:17.541 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T03:04:17.541 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T03:04:17.542 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T03:04:17.616 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T03:04:17.620 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T03:04:17.655 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:04:17.666 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.60'), ('mon.b', '[v2:172.21.15.60:3301,v1:172.21.15.60:6790]'), ('mon.c', '[v2:172.21.15.60:3302,v1:172.21.15.60:6791]')] 2024-09-15T03:04:17.667 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.60,[v2:172.21.15.60:3301,v1:172.21.15.60:6790],[v2:172.21.15.60:3302,v1:172.21.15.60:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-15T03:04:17.668 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:17.668 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:04:17.675 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --addv b '[v2:172.21.15.60:3301,v1:172.21.15.60:6790]' --addv c '[v2:172.21.15.60:3302,v1:172.21.15.60:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:04:17.758 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:04:17.758 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = octopus 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:fsid c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2024-09-15T03:04:17.756422+0000 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-09-15T03:04:17.756422+0000 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 15 (octopus) 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 1 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:1: [v2:172.21.15.60:3301/0,v1:172.21.15.60:6790/0] mon.b 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:2: [v2:172.21.15.60:3302/0,v1:172.21.15.60:6791/0] mon.c 2024-09-15T03:04:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T03:04:17.761 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:04:17.766 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99... 2024-09-15T03:04:17.767 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-09-15T03:04:17.803 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-09-15T03:04:17.804 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.805 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "172.21.15.60,[v2:172.21.15.60:3301,v1:172.21.15.60:6790],[v2:172.21.15.60:3302,v1:172.21.15.60:6791]" 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: mon election default strategy = 1 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 5000 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: ms type = async 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.806 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.807 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression algorithm = zlib 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore compression mode = aggressive 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-09-15T03:04:17.808 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore zero block detection = True 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 20 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 20 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 10 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify cached snaps = True 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify missing on start = True 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock override recovery settings = True 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-09-15T03:04:17.809 INFO:teuthology.orchestra.run.smithi060.stdout: osd op queue = debug_random 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: osd op queue cut off = debug_random 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-15T03:04:17.810 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.811 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: mon scrub interval = 300 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-09-15T03:04:17.812 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:04:17.813 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:04:17.813 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-09-15T03:04:17.813 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2024-09-15T03:04:17.813 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2024-09-15T03:04:17.814 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T03:04:17.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T03:04:17.907 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T03:04:17.907 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:17.907 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T03:04:17.915 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:17.915 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T03:04:17.920 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:04:17.920 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:17.920 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:04:17.920 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:04:17.945 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:17.945 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:04:17.950 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T03:04:17.951 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T03:04:17.951 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:04:18.043 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:04:18.046 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T03:04:18.046 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T03:04:18.046 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T03:04:18.046 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-15T03:04:18.046 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T03:04:18.057 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:04:18.057 INFO:tasks.ceph:role: osd.0 2024-09-15T03:04:18.057 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:04:18.057 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:04:18.101 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:04:18.101 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:04:18.101 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:04:18.102 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 2024-09-15T03:04:18.102 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:04:18.102 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:04:18.102 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:04:18.102 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:04:18.102 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:04:18.102 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:04:18.103 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-15T03:04:18.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T03:04:18.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T03:04:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:04:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:04:18.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T03:04:18.185 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:04:18.185 INFO:tasks.ceph:role: osd.1 2024-09-15T03:04:18.185 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:04:18.185 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:04:18.271 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:04:18.271 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:04:18.271 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:04:18.271 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 2024-09-15T03:04:18.271 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:04:18.272 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:04:18.272 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:04:18.272 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:04:18.272 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:04:18.272 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:04:18.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-15T03:04:18.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T03:04:18.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T03:04:18.303 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:04:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:04:18.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T03:04:18.360 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:04:18.361 INFO:tasks.ceph:role: osd.2 2024-09-15T03:04:18.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:04:18.361 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:04:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:04:18.403 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:04:18.403 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:04:18.403 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 2024-09-15T03:04:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:04:18.404 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:04:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:04:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:04:18.404 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:04:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:04:18.405 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-15T03:04:18.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T03:04:18.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T03:04:18.473 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:04:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:04:18.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T03:04:18.485 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T03:04:18.486 INFO:tasks.ceph:role: osd.3 2024-09-15T03:04:18.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:04:18.486 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T03:04:18.528 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:04:18.528 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:04:18.528 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:04:18.528 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 2024-09-15T03:04:18.529 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:04:18.529 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:04:18.529 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:04:18.529 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:04:18.529 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:04:18.529 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:04:18.530 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-15T03:04:18.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-15T03:04:18.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T03:04:18.605 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:04:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:04:18.606 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:04:18.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:18.674+0000 7f12a8949540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T03:04:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:18.674+0000 7f12a8949540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T03:04:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:18.674+0000 7f12a8949540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:04:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:18.674+0000 7f12a8949540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:04:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:18.674+0000 7f12a8949540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:04:18.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:18.674+0000 7f12a8949540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T03:04:19.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:19.222+0000 7f12a8949540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:20.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:20.002+0000 7f12a8949540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:20.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:20.786+0000 7f12a8949540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:21.650 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T03:04:21.666 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:04:21.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:21.742+0000 7fcb07e38540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T03:04:21.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:21.742+0000 7fcb07e38540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T03:04:21.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:21.742+0000 7fcb07e38540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:04:21.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:21.742+0000 7fcb07e38540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:04:21.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:21.742+0000 7fcb07e38540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:04:21.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:21.746+0000 7fcb07e38540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T03:04:22.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:22.294+0000 7fcb07e38540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:23.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:23.078+0000 7fcb07e38540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:23.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:23.858+0000 7fcb07e38540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:24.725 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T03:04:24.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:04:24.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:24.774+0000 7f2bd31c5540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T03:04:24.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:24.774+0000 7f2bd31c5540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T03:04:24.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:24.774+0000 7f2bd31c5540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:04:24.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:24.774+0000 7f2bd31c5540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:04:24.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:24.774+0000 7f2bd31c5540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:04:24.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:24.774+0000 7f2bd31c5540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T03:04:25.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:25.330+0000 7f2bd31c5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:26.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:26.114+0000 7f2bd31c5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:26.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:26.902+0000 7f2bd31c5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:27.749 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T03:04:27.804 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:04:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:27.838+0000 7f5ce6ca5540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T03:04:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:27.838+0000 7f5ce6ca5540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T03:04:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:27.838+0000 7f5ce6ca5540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:04:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:27.838+0000 7f5ce6ca5540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:04:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:27.838+0000 7f5ce6ca5540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:04:27.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:27.838+0000 7f5ce6ca5540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T03:04:28.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:28.390+0000 7f5ce6ca5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:29.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:29.178+0000 7f5ce6ca5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:29.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:29.970+0000 7f5ce6ca5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:04:30.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T03:04:30.833 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T03:04:30.834 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:30.834 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T03:04:30.893 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:30.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T03:04:30.902 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:30.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T03:04:30.912 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:30.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T03:04:30.924 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:30.924 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T03:04:30.979 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T03:04:30.979 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCyTuZm6SuGAhAAJpuVUbwVBMY/QneIVZCSEQ== 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQCyTuZmg/lmKBAAOdxJWG4VR9w/dI66EF6dYQ== 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC1TuZmTJOJLBAAbKLnuSFMu/OhFLY4nqIs0g== 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC4TuZmclFeLhAAyf17utEYIHjsxfm6HO80dQ== 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2024-09-15T03:04:30.995 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC7TuZmY7kcMhAASv4d/dkZuBi45mU7pjf/GA== 2024-09-15T03:04:30.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:04:31.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:04:31.125 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:04:31.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:04:31.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:04:31.298 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T03:04:31.298 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T03:04:31.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:04:31.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T03:04:31.430 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T03:04:31.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:04:31.497 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T03:04:31.510 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T03:04:31.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:04:31.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T03:04:31.590 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:04:31.597 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T03:04:31.597 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:04:31.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:04:31.599 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:04:31.599 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:04:31.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:04:31.601 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:04:31.602 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:04:31.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:04:31.604 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:04:31.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph config set mgr mgr_pool false --force 2024-09-15T03:04:37.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:04:37.914+0000 7fe8150d5700 0 --2- 172.21.15.60:0/1686360476 >> v2:172.21.15.60:3300/0 conn(0x7fe7f40063e0 0x7fe7f4008820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:04:38.022 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T03:04:38.023 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T03:04:38.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T03:04:38.025 INFO:tasks.ceph.mgr.x:Started 2024-09-15T03:04:38.026 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T03:04:38.026 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T03:04:38.218 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-09-15T03:04:38.227 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T03:04:38.228 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:38.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T03:04:38.237 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:38.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T03:04:38.247 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:38.247 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T03:04:38.260 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-15T03:04:38.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T03:04:38.311 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:38.306+0000 7f9aff9b8000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T03:04:38.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 6156de49-3ab1-449c-a1c6-08954b367614 0 2024-09-15T03:04:38.514 INFO:teuthology.orchestra.run.smithi060.stdout:0 2024-09-15T03:04:38.526 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 6cddb0b6-6f15-4fb6-a24b-4741e438204b 1 2024-09-15T03:04:38.738 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-09-15T03:04:38.753 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 733d9ab1-4b03-48a7-8953-0ab0e40ef743 2 2024-09-15T03:04:38.951 INFO:teuthology.orchestra.run.smithi060.stdout:2 2024-09-15T03:04:38.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new baf7b285-0b3b-402f-bf3b-2fc9316f8b3a 3 2024-09-15T03:04:39.060 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:39.054+0000 7f9aff9b8000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T03:04:39.120 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:39.114+0000 7f9aff9b8000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T03:04:39.164 INFO:teuthology.orchestra.run.smithi060.stdout:3 2024-09-15T03:04:39.176 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T03:04:39.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T03:04:39.178 INFO:tasks.ceph.osd.0:Started 2024-09-15T03:04:39.178 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T03:04:39.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T03:04:39.181 INFO:tasks.ceph.osd.1:Started 2024-09-15T03:04:39.182 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T03:04:39.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T03:04:39.184 INFO:tasks.ceph.osd.2:Started 2024-09-15T03:04:39.184 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T03:04:39.184 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T03:04:39.186 INFO:tasks.ceph.osd.3:Started 2024-09-15T03:04:39.187 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:04:39.273 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:39.266+0000 7f9aff9b8000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:04:39.374 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:39.374 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":6,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:04:39.157850+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:04:39.391 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T03:04:39.391 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T03:04:39.522 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:39.514+0000 7f9aff9b8000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T03:04:39.564 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:04:39.558+0000 7f6366776540 -1 Falling back to public interface 2024-09-15T03:04:39.571 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:04:39.566+0000 7f08a42e1540 -1 Falling back to public interface 2024-09-15T03:04:39.577 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:39.570+0000 7f9aff9b8000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T03:04:39.583 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:04:39.578+0000 7f90a0f38540 -1 Falling back to public interface 2024-09-15T03:04:39.780 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:39.774+0000 7f9aff9b8000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T03:04:39.794 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:04:39.820 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-09-15T03:04:39.814+0000 7f9bdbf46540 -1 Falling back to public interface 2024-09-15T03:04:39.847 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:39.842+0000 7f9aff9b8000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T03:04:39.923 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:39.918+0000 7f9aff9b8000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T03:04:39.966 INFO:teuthology.misc.health.smithi060.stdout: 2024-09-15T03:04:39.967 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":6,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:04:39.157850+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:04:39.981 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-15T03:04:40.372 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:40.366+0000 7f9aff9b8000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T03:04:40.499 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:40.494+0000 7f9aff9b8000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T03:04:40.825 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:40.818+0000 7f9aff9b8000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T03:04:41.001 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:40.994+0000 7f9aff9b8000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:04:41.070 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:41.066+0000 7f9aff9b8000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T03:04:41.125 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:41.118+0000 7f9aff9b8000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T03:04:41.184 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:41.178+0000 7f9aff9b8000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T03:04:41.358 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:41.354+0000 7f9aff9b8000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T03:04:41.526 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:41.522+0000 7f9aff9b8000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T03:04:41.595 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:41.590+0000 7f9aff9b8000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T03:04:41.761 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:41.754+0000 7f9aff9b8000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T03:04:41.834 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:41.826+0000 7f9aff9b8000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T03:04:42.589 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:42.582+0000 7f9aff9b8000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T03:04:43.104 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:04:43.098+0000 7f6366776540 -1 osd.0 0 log_to_monitors true 2024-09-15T03:04:43.339 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:04:43.334+0000 7f08a42e1540 -1 osd.1 0 log_to_monitors true 2024-09-15T03:04:43.380 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:04:43.374+0000 7f90a0f38540 -1 osd.3 0 log_to_monitors true 2024-09-15T03:04:43.626 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:43.622+0000 7f9ae8a2f700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T03:04:43.633 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-09-15T03:04:43.630+0000 7f9bdbf46540 -1 osd.2 0 log_to_monitors true 2024-09-15T03:04:44.625 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-15T03:04:44.618+0000 7f9ae8a2f700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T03:04:44.689 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:04:44.682+0000 7f6361f16700 -1 osd.0 0 waiting for initial osdmap 2024-09-15T03:04:44.693 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:04:44.686+0000 7f089fa81700 -1 osd.1 0 waiting for initial osdmap 2024-09-15T03:04:44.693 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-09-15T03:04:44.686+0000 7f9bd76e6700 -1 osd.2 0 waiting for initial osdmap 2024-09-15T03:04:44.697 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:04:44.690+0000 7f909defd700 -1 osd.3 0 waiting for initial osdmap 2024-09-15T03:04:44.698 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:04:44.694+0000 7f089a027700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:04:44.698 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:04:44.694+0000 7f635c4bc700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:04:44.699 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-09-15T03:04:44.694+0000 7f9bd148b700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:04:44.701 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:04:44.694+0000 7f9096c7e700 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:04:46.383 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:04:46.701 INFO:teuthology.misc.health.smithi060.stdout: 2024-09-15T03:04:46.702 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":9,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:04:45.685742+0000","last_up_change":"2024-09-15T03:04:45.685742+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":22281},{"type":"v1","addr":"172.21.15.60:6801","nonce":22281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":22281},{"type":"v1","addr":"172.21.15.60:6803","nonce":22281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":22281},{"type":"v1","addr":"172.21.15.60:6807","nonce":22281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":22281},{"type":"v1","addr":"172.21.15.60:6805","nonce":22281}]},"public_addr":"172.21.15.60:6801/22281","cluster_addr":"172.21.15.60:6803/22281","heartbeat_back_addr":"172.21.15.60:6807/22281","heartbeat_front_addr":"172.21.15.60:6805/22281","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":22282},{"type":"v1","addr":"172.21.15.60:6809","nonce":22282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":22282},{"type":"v1","addr":"172.21.15.60:6811","nonce":22282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":22282},{"type":"v1","addr":"172.21.15.60:6815","nonce":22282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":22282},{"type":"v1","addr":"172.21.15.60:6813","nonce":22282}]},"public_addr":"172.21.15.60:6809/22282","cluster_addr":"172.21.15.60:6811/22282","heartbeat_back_addr":"172.21.15.60:6815/22282","heartbeat_front_addr":"172.21.15.60:6813/22282","state":["exists","up"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":22284},{"type":"v1","addr":"172.21.15.60:6817","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":22284},{"type":"v1","addr":"172.21.15.60:6819","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":22284},{"type":"v1","addr":"172.21.15.60:6823","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":22284},{"type":"v1","addr":"172.21.15.60:6821","nonce":22284}]},"public_addr":"172.21.15.60:6817/22284","cluster_addr":"172.21.15.60:6819/22284","heartbeat_back_addr":"172.21.15.60:6823/22284","heartbeat_front_addr":"172.21.15.60:6821/22284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:04:46.715 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-15T03:04:46.715 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T03:04:46.716 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T03:04:46.716 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T03:04:46.716 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T03:04:46.716 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T03:04:47.045 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:47.059 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"flags":0,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":22104},{"type":"v1","addr":"172.21.15.60:6833","nonce":22104}]},"active_addr":"172.21.15.60:6833/22104","active_change":"2024-09-15T03:04:42.606867+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2591655472}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":3725852739}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2916397236}]}]} 2024-09-15T03:04:47.062 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T03:04:47.062 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T03:04:47.063 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:04:47.353 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:47.353 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:04:46.698209+0000","last_up_change":"2024-09-15T03:04:45.685742+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":22281},{"type":"v1","addr":"172.21.15.60:6801","nonce":22281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":22281},{"type":"v1","addr":"172.21.15.60:6803","nonce":22281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":22281},{"type":"v1","addr":"172.21.15.60:6807","nonce":22281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":22281},{"type":"v1","addr":"172.21.15.60:6805","nonce":22281}]},"public_addr":"172.21.15.60:6801/22281","cluster_addr":"172.21.15.60:6803/22281","heartbeat_back_addr":"172.21.15.60:6807/22281","heartbeat_front_addr":"172.21.15.60:6805/22281","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":22282},{"type":"v1","addr":"172.21.15.60:6809","nonce":22282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":22282},{"type":"v1","addr":"172.21.15.60:6811","nonce":22282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":22282},{"type":"v1","addr":"172.21.15.60:6815","nonce":22282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":22282},{"type":"v1","addr":"172.21.15.60:6813","nonce":22282}]},"public_addr":"172.21.15.60:6809/22282","cluster_addr":"172.21.15.60:6811/22282","heartbeat_back_addr":"172.21.15.60:6815/22282","heartbeat_front_addr":"172.21.15.60:6813/22282","state":["exists","up"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":22284},{"type":"v1","addr":"172.21.15.60:6817","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":22284},{"type":"v1","addr":"172.21.15.60:6819","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":22284},{"type":"v1","addr":"172.21.15.60:6823","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":22284},{"type":"v1","addr":"172.21.15.60:6821","nonce":22284}]},"public_addr":"172.21.15.60:6817/22284","cluster_addr":"172.21.15.60:6819/22284","heartbeat_back_addr":"172.21.15.60:6823/22284","heartbeat_front_addr":"172.21.15.60:6821/22284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:04:47.367 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T03:04:47.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:04:47.653 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:47.653 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:04:46.698209+0000","last_up_change":"2024-09-15T03:04:45.685742+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":22281},{"type":"v1","addr":"172.21.15.60:6801","nonce":22281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":22281},{"type":"v1","addr":"172.21.15.60:6803","nonce":22281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":22281},{"type":"v1","addr":"172.21.15.60:6807","nonce":22281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":22281},{"type":"v1","addr":"172.21.15.60:6805","nonce":22281}]},"public_addr":"172.21.15.60:6801/22281","cluster_addr":"172.21.15.60:6803/22281","heartbeat_back_addr":"172.21.15.60:6807/22281","heartbeat_front_addr":"172.21.15.60:6805/22281","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":22282},{"type":"v1","addr":"172.21.15.60:6809","nonce":22282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":22282},{"type":"v1","addr":"172.21.15.60:6811","nonce":22282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":22282},{"type":"v1","addr":"172.21.15.60:6815","nonce":22282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":22282},{"type":"v1","addr":"172.21.15.60:6813","nonce":22282}]},"public_addr":"172.21.15.60:6809/22282","cluster_addr":"172.21.15.60:6811/22282","heartbeat_back_addr":"172.21.15.60:6815/22282","heartbeat_front_addr":"172.21.15.60:6813/22282","state":["exists","up"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":22284},{"type":"v1","addr":"172.21.15.60:6817","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":22284},{"type":"v1","addr":"172.21.15.60:6819","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":22284},{"type":"v1","addr":"172.21.15.60:6823","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":22284},{"type":"v1","addr":"172.21.15.60:6821","nonce":22284}]},"public_addr":"172.21.15.60:6817/22284","cluster_addr":"172.21.15.60:6819/22284","heartbeat_back_addr":"172.21.15.60:6823/22284","heartbeat_front_addr":"172.21.15.60:6821/22284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:04:47.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:04:47.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:04:47.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:04:47.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:04:47.812 INFO:teuthology.orchestra.run.smithi060.stdout:38654705667 2024-09-15T03:04:47.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:04:47.823 INFO:teuthology.orchestra.run.smithi060.stdout:38654705666 2024-09-15T03:04:47.824 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:04:47.824 INFO:teuthology.orchestra.run.smithi060.stdout:38654705666 2024-09-15T03:04:47.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:04:47.831 INFO:teuthology.orchestra.run.smithi060.stdout:38654705666 2024-09-15T03:04:47.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:04:48.091 INFO:teuthology.orchestra.run.smithi060.stdout:38654705666 2024-09-15T03:04:48.104 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.0 2024-09-15T03:04:48.149 INFO:teuthology.orchestra.run.smithi060.stdout:38654705665 2024-09-15T03:04:48.150 INFO:teuthology.orchestra.run.smithi060.stdout:38654705665 2024-09-15T03:04:48.151 INFO:teuthology.orchestra.run.smithi060.stdout:38654705665 2024-09-15T03:04:48.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.1 2024-09-15T03:04:48.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.2 2024-09-15T03:04:48.168 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.3 2024-09-15T03:04:49.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:04:49.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:04:49.166 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:04:49.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:04:49.443 INFO:teuthology.orchestra.run.smithi060.stdout:38654705667 2024-09-15T03:04:49.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.0 2024-09-15T03:04:49.456 DEBUG:teuthology.parallel:result is None 2024-09-15T03:04:49.459 INFO:teuthology.orchestra.run.smithi060.stdout:38654705666 2024-09-15T03:04:49.472 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.2 2024-09-15T03:04:49.472 DEBUG:teuthology.parallel:result is None 2024-09-15T03:04:49.498 INFO:teuthology.orchestra.run.smithi060.stdout:38654705666 2024-09-15T03:04:49.501 INFO:teuthology.orchestra.run.smithi060.stdout:38654705666 2024-09-15T03:04:49.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.1 2024-09-15T03:04:49.512 DEBUG:teuthology.parallel:result is None 2024-09-15T03:04:49.520 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.3 2024-09-15T03:04:49.520 DEBUG:teuthology.parallel:result is None 2024-09-15T03:04:49.520 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:04:49.520 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:04:49.803 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:49.807 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:04:49.817 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-09-15T03:04:48.619136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":1189760,"kb_used_data":336,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297600,"statfs":{"total":386547056640,"available":385328742400,"internally_reserved":0,"allocated":344064,"data_stored":74618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-09-15T03:04:49.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:04:50.112 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:04:50.125 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":8,"stamp":"2024-09-15T03:04:48.619136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":1189760,"kb_used_data":336,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297600,"statfs":{"total":386547056640,"available":385328742400,"internally_reserved":0,"allocated":344064,"data_stored":74618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-09-15T03:04:50.126 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:04:50.126 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T03:04:50.126 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T03:04:50.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T03:04:50.445 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:50.445 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T03:04:50.459 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T03:04:50.459 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2024-09-15T03:04:50.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:04:50.766 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:50.767 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:04:46.698209+0000","last_up_change":"2024-09-15T03:04:45.685742+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":22281},{"type":"v1","addr":"172.21.15.60:6801","nonce":22281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":22281},{"type":"v1","addr":"172.21.15.60:6803","nonce":22281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":22281},{"type":"v1","addr":"172.21.15.60:6807","nonce":22281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":22281},{"type":"v1","addr":"172.21.15.60:6805","nonce":22281}]},"public_addr":"172.21.15.60:6801/22281","cluster_addr":"172.21.15.60:6803/22281","heartbeat_back_addr":"172.21.15.60:6807/22281","heartbeat_front_addr":"172.21.15.60:6805/22281","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":22282},{"type":"v1","addr":"172.21.15.60:6809","nonce":22282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":22282},{"type":"v1","addr":"172.21.15.60:6811","nonce":22282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":22282},{"type":"v1","addr":"172.21.15.60:6815","nonce":22282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":22282},{"type":"v1","addr":"172.21.15.60:6813","nonce":22282}]},"public_addr":"172.21.15.60:6809/22282","cluster_addr":"172.21.15.60:6811/22282","heartbeat_back_addr":"172.21.15.60:6815/22282","heartbeat_front_addr":"172.21.15.60:6813/22282","state":["exists","up"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":22284},{"type":"v1","addr":"172.21.15.60:6817","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":22284},{"type":"v1","addr":"172.21.15.60:6819","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":22284},{"type":"v1","addr":"172.21.15.60:6823","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":22284},{"type":"v1","addr":"172.21.15.60:6821","nonce":22284}]},"public_addr":"172.21.15.60:6817/22284","cluster_addr":"172.21.15.60:6819/22284","heartbeat_back_addr":"172.21.15.60:6823/22284","heartbeat_front_addr":"172.21.15.60:6821/22284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:04:50.780 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2024-09-15T03:04:50.780 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-15T03:04:50.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:04:51.062 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:04:51.076 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-09-15T03:04:50.619617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":1189760,"kb_used_data":336,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297600,"statfs":{"total":386547056640,"available":385328742400,"internally_reserved":0,"allocated":344064,"data_stored":74618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-09-15T03:04:51.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:04:51.359 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:04:51.362 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:04:51.373 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-09-15T03:04:50.619617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":1189760,"kb_used_data":336,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297600,"statfs":{"total":386547056640,"available":385328742400,"internally_reserved":0,"allocated":344064,"data_stored":74618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.016}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297436,"kb_used_data":80,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":81920,"data_stored":17144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":297444,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074396,"statfs":{"total":96636764160,"available":96332181504,"internally_reserved":0,"allocated":90112,"data_stored":20165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-09-15T03:04:51.374 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-15T03:04:51.374 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2024-09-15T03:04:51.748 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2024-09-15T03:04:51.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2024-09-15T03:04:53.758 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'unique_pool_0' created 2024-09-15T03:04:53.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-09-15T03:04:54.760 INFO:teuthology.orchestra.run.smithi060.stderr:set pool 1 min_size to 2 2024-09-15T03:04:54.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-15T03:04:55.778 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-15T03:04:56.793 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2024-09-15T03:04:56.796 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2024-09-15T03:04:56.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:04:56.850 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:04:56.998 INFO:teuthology.orchestra.run.smithi060.stdout:38654705669 2024-09-15T03:04:56.998 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:04:57.006 INFO:teuthology.orchestra.run.smithi060.stdout:38654705670 2024-09-15T03:04:57.006 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:04:57.286 INFO:teuthology.orchestra.run.smithi060.stdout:38654705667 2024-09-15T03:04:57.291 INFO:teuthology.orchestra.run.smithi060.stdout:38654705668 2024-09-15T03:04:57.307 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705667 for osd.1 2024-09-15T03:04:57.308 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705668 for osd.0 2024-09-15T03:04:58.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:04:58.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:04:58.603 INFO:teuthology.orchestra.run.smithi060.stdout:38654705668 2024-09-15T03:04:58.609 INFO:teuthology.orchestra.run.smithi060.stdout:38654705667 2024-09-15T03:04:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705667 for osd.1 2024-09-15T03:04:58.623 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705668 for osd.0 2024-09-15T03:04:59.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:04:59.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:04:59.912 INFO:teuthology.orchestra.run.smithi060.stdout:38654705669 2024-09-15T03:04:59.919 INFO:teuthology.orchestra.run.smithi060.stdout:38654705670 2024-09-15T03:04:59.927 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705669 got 38654705669 for osd.1 2024-09-15T03:04:59.927 DEBUG:teuthology.parallel:result is None 2024-09-15T03:04:59.933 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705670 got 38654705670 for osd.0 2024-09-15T03:04:59.934 DEBUG:teuthology.parallel:result is None 2024-09-15T03:04:59.934 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-15T03:04:59.934 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:05:00.219 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:00.222 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:05:00.234 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-15T03:04:58.621542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189936,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297424,"statfs":{"total":386547056640,"available":385328562176,"internally_reserved":0,"allocated":507904,"data_stored":148214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.753985+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:05:05.130594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.777808+0000","last_change":"2024-09-15T03:04:55.775319+0000","last_active":"2024-09-15T03:04:55.777808+0000","last_peered":"2024-09-15T03:04:55.777808+0000","last_clean":"2024-09-15T03:04:55.777808+0000","last_became_active":"2024-09-15T03:04:55.774882+0000","last_became_peered":"2024-09-15T03:04:55.774882+0000","last_unstale":"2024-09-15T03:04:55.777808+0000","last_undegraded":"2024-09-15T03:04:55.777808+0000","last_fullsized":"2024-09-15T03:04:55.777808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:26:48.042045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.769265+0000","last_change":"2024-09-15T03:04:55.769265+0000","last_active":"2024-09-15T03:04:55.769265+0000","last_peered":"2024-09-15T03:04:55.769265+0000","last_clean":"2024-09-15T03:04:55.769265+0000","last_became_active":"2024-09-15T03:04:55.768372+0000","last_became_peered":"2024-09-15T03:04:55.768372+0000","last_unstale":"2024-09-15T03:04:55.769265+0000","last_undegraded":"2024-09-15T03:04:55.769265+0000","last_fullsized":"2024-09-15T03:04:55.769265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:51.682552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778410+0000","last_change":"2024-09-15T03:04:55.777392+0000","last_active":"2024-09-15T03:04:55.778410+0000","last_peered":"2024-09-15T03:04:55.778410+0000","last_clean":"2024-09-15T03:04:55.778410+0000","last_became_active":"2024-09-15T03:04:55.767816+0000","last_became_peered":"2024-09-15T03:04:55.767816+0000","last_unstale":"2024-09-15T03:04:55.778410+0000","last_undegraded":"2024-09-15T03:04:55.778410+0000","last_fullsized":"2024-09-15T03:04:55.778410+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:49:25.460875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.753047+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:36:45.108947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.752703+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:22:44.914555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.771342+0000","last_change":"2024-09-15T03:04:55.770349+0000","last_active":"2024-09-15T03:04:55.771342+0000","last_peered":"2024-09-15T03:04:55.771342+0000","last_clean":"2024-09-15T03:04:55.771342+0000","last_became_active":"2024-09-15T03:04:55.769687+0000","last_became_peered":"2024-09-15T03:04:55.769687+0000","last_unstale":"2024-09-15T03:04:55.771342+0000","last_undegraded":"2024-09-15T03:04:55.771342+0000","last_fullsized":"2024-09-15T03:04:55.771342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:39:39.315986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778132+0000","last_change":"2024-09-15T03:04:55.775593+0000","last_active":"2024-09-15T03:04:55.778132+0000","last_peered":"2024-09-15T03:04:55.778132+0000","last_clean":"2024-09-15T03:04:55.778132+0000","last_became_active":"2024-09-15T03:04:55.774855+0000","last_became_peered":"2024-09-15T03:04:55.774855+0000","last_unstale":"2024-09-15T03:04:55.778132+0000","last_undegraded":"2024-09-15T03:04:55.778132+0000","last_fullsized":"2024-09-15T03:04:55.778132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:14:02.439822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.752135+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:22.230150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.763931+0000","last_change":"2024-09-15T03:04:55.763931+0000","last_active":"2024-09-15T03:04:55.763931+0000","last_peered":"2024-09-15T03:04:55.763931+0000","last_clean":"2024-09-15T03:04:55.763931+0000","last_became_active":"2024-09-15T03:04:55.763333+0000","last_became_peered":"2024-09-15T03:04:55.763333+0000","last_unstale":"2024-09-15T03:04:55.763931+0000","last_undegraded":"2024-09-15T03:04:55.763931+0000","last_fullsized":"2024-09-15T03:04:55.763931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:51:05.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.751529+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:36:59.232182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778371+0000","last_change":"2024-09-15T03:04:55.777926+0000","last_active":"2024-09-15T03:04:55.778371+0000","last_peered":"2024-09-15T03:04:55.778371+0000","last_clean":"2024-09-15T03:04:55.778371+0000","last_became_active":"2024-09-15T03:04:55.766750+0000","last_became_peered":"2024-09-15T03:04:55.766750+0000","last_unstale":"2024-09-15T03:04:55.778371+0000","last_undegraded":"2024-09-15T03:04:55.778371+0000","last_fullsized":"2024-09-15T03:04:55.778371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:47:19.790201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778389+0000","last_change":"2024-09-15T03:04:55.777093+0000","last_active":"2024-09-15T03:04:55.778389+0000","last_peered":"2024-09-15T03:04:55.778389+0000","last_clean":"2024-09-15T03:04:55.778389+0000","last_became_active":"2024-09-15T03:04:55.767200+0000","last_became_peered":"2024-09-15T03:04:55.767200+0000","last_unstale":"2024-09-15T03:04:55.778389+0000","last_undegraded":"2024-09-15T03:04:55.778389+0000","last_fullsized":"2024-09-15T03:04:55.778389+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:38.492407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.780670+0000","last_change":"2024-09-15T03:04:55.778257+0000","last_active":"2024-09-15T03:04:55.780670+0000","last_peered":"2024-09-15T03:04:55.780670+0000","last_clean":"2024-09-15T03:04:55.780670+0000","last_became_active":"2024-09-15T03:04:55.775400+0000","last_became_peered":"2024-09-15T03:04:55.775400+0000","last_unstale":"2024-09-15T03:04:55.780670+0000","last_undegraded":"2024-09-15T03:04:55.780670+0000","last_fullsized":"2024-09-15T03:04:55.780670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:02:00.435436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.777801+0000","last_change":"2024-09-15T03:04:55.774534+0000","last_active":"2024-09-15T03:04:55.777801+0000","last_peered":"2024-09-15T03:04:55.777801+0000","last_clean":"2024-09-15T03:04:55.777801+0000","last_became_active":"2024-09-15T03:04:55.766007+0000","last_became_peered":"2024-09-15T03:04:55.766007+0000","last_unstale":"2024-09-15T03:04:55.777801+0000","last_undegraded":"2024-09-15T03:04:55.777801+0000","last_fullsized":"2024-09-15T03:04:55.777801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:50:12.599469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778159+0000","last_change":"2024-09-15T03:04:55.777104+0000","last_active":"2024-09-15T03:04:55.778159+0000","last_peered":"2024-09-15T03:04:55.778159+0000","last_clean":"2024-09-15T03:04:55.778159+0000","last_became_active":"2024-09-15T03:04:55.774292+0000","last_became_peered":"2024-09-15T03:04:55.774292+0000","last_unstale":"2024-09-15T03:04:55.778159+0000","last_undegraded":"2024-09-15T03:04:55.778159+0000","last_fullsized":"2024-09-15T03:04:55.778159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:18.102228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":35104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":35104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":39003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":39003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:05:00.235 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:05:00.520 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:00.523 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:05:00.535 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-15T03:04:58.621542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189936,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297424,"statfs":{"total":386547056640,"available":385328562176,"internally_reserved":0,"allocated":507904,"data_stored":148214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.753985+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:05:05.130594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.777808+0000","last_change":"2024-09-15T03:04:55.775319+0000","last_active":"2024-09-15T03:04:55.777808+0000","last_peered":"2024-09-15T03:04:55.777808+0000","last_clean":"2024-09-15T03:04:55.777808+0000","last_became_active":"2024-09-15T03:04:55.774882+0000","last_became_peered":"2024-09-15T03:04:55.774882+0000","last_unstale":"2024-09-15T03:04:55.777808+0000","last_undegraded":"2024-09-15T03:04:55.777808+0000","last_fullsized":"2024-09-15T03:04:55.777808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:26:48.042045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.769265+0000","last_change":"2024-09-15T03:04:55.769265+0000","last_active":"2024-09-15T03:04:55.769265+0000","last_peered":"2024-09-15T03:04:55.769265+0000","last_clean":"2024-09-15T03:04:55.769265+0000","last_became_active":"2024-09-15T03:04:55.768372+0000","last_became_peered":"2024-09-15T03:04:55.768372+0000","last_unstale":"2024-09-15T03:04:55.769265+0000","last_undegraded":"2024-09-15T03:04:55.769265+0000","last_fullsized":"2024-09-15T03:04:55.769265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:51.682552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778410+0000","last_change":"2024-09-15T03:04:55.777392+0000","last_active":"2024-09-15T03:04:55.778410+0000","last_peered":"2024-09-15T03:04:55.778410+0000","last_clean":"2024-09-15T03:04:55.778410+0000","last_became_active":"2024-09-15T03:04:55.767816+0000","last_became_peered":"2024-09-15T03:04:55.767816+0000","last_unstale":"2024-09-15T03:04:55.778410+0000","last_undegraded":"2024-09-15T03:04:55.778410+0000","last_fullsized":"2024-09-15T03:04:55.778410+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:49:25.460875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.753047+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:36:45.108947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.752703+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:22:44.914555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.771342+0000","last_change":"2024-09-15T03:04:55.770349+0000","last_active":"2024-09-15T03:04:55.771342+0000","last_peered":"2024-09-15T03:04:55.771342+0000","last_clean":"2024-09-15T03:04:55.771342+0000","last_became_active":"2024-09-15T03:04:55.769687+0000","last_became_peered":"2024-09-15T03:04:55.769687+0000","last_unstale":"2024-09-15T03:04:55.771342+0000","last_undegraded":"2024-09-15T03:04:55.771342+0000","last_fullsized":"2024-09-15T03:04:55.771342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:39:39.315986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778132+0000","last_change":"2024-09-15T03:04:55.775593+0000","last_active":"2024-09-15T03:04:55.778132+0000","last_peered":"2024-09-15T03:04:55.778132+0000","last_clean":"2024-09-15T03:04:55.778132+0000","last_became_active":"2024-09-15T03:04:55.774855+0000","last_became_peered":"2024-09-15T03:04:55.774855+0000","last_unstale":"2024-09-15T03:04:55.778132+0000","last_undegraded":"2024-09-15T03:04:55.778132+0000","last_fullsized":"2024-09-15T03:04:55.778132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:14:02.439822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.752135+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:22.230150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.763931+0000","last_change":"2024-09-15T03:04:55.763931+0000","last_active":"2024-09-15T03:04:55.763931+0000","last_peered":"2024-09-15T03:04:55.763931+0000","last_clean":"2024-09-15T03:04:55.763931+0000","last_became_active":"2024-09-15T03:04:55.763333+0000","last_became_peered":"2024-09-15T03:04:55.763333+0000","last_unstale":"2024-09-15T03:04:55.763931+0000","last_undegraded":"2024-09-15T03:04:55.763931+0000","last_fullsized":"2024-09-15T03:04:55.763931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:51:05.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.751529+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:36:59.232182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778371+0000","last_change":"2024-09-15T03:04:55.777926+0000","last_active":"2024-09-15T03:04:55.778371+0000","last_peered":"2024-09-15T03:04:55.778371+0000","last_clean":"2024-09-15T03:04:55.778371+0000","last_became_active":"2024-09-15T03:04:55.766750+0000","last_became_peered":"2024-09-15T03:04:55.766750+0000","last_unstale":"2024-09-15T03:04:55.778371+0000","last_undegraded":"2024-09-15T03:04:55.778371+0000","last_fullsized":"2024-09-15T03:04:55.778371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:47:19.790201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778389+0000","last_change":"2024-09-15T03:04:55.777093+0000","last_active":"2024-09-15T03:04:55.778389+0000","last_peered":"2024-09-15T03:04:55.778389+0000","last_clean":"2024-09-15T03:04:55.778389+0000","last_became_active":"2024-09-15T03:04:55.767200+0000","last_became_peered":"2024-09-15T03:04:55.767200+0000","last_unstale":"2024-09-15T03:04:55.778389+0000","last_undegraded":"2024-09-15T03:04:55.778389+0000","last_fullsized":"2024-09-15T03:04:55.778389+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:38.492407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.780670+0000","last_change":"2024-09-15T03:04:55.778257+0000","last_active":"2024-09-15T03:04:55.780670+0000","last_peered":"2024-09-15T03:04:55.780670+0000","last_clean":"2024-09-15T03:04:55.780670+0000","last_became_active":"2024-09-15T03:04:55.775400+0000","last_became_peered":"2024-09-15T03:04:55.775400+0000","last_unstale":"2024-09-15T03:04:55.780670+0000","last_undegraded":"2024-09-15T03:04:55.780670+0000","last_fullsized":"2024-09-15T03:04:55.780670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:02:00.435436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.777801+0000","last_change":"2024-09-15T03:04:55.774534+0000","last_active":"2024-09-15T03:04:55.777801+0000","last_peered":"2024-09-15T03:04:55.777801+0000","last_clean":"2024-09-15T03:04:55.777801+0000","last_became_active":"2024-09-15T03:04:55.766007+0000","last_became_peered":"2024-09-15T03:04:55.766007+0000","last_unstale":"2024-09-15T03:04:55.777801+0000","last_undegraded":"2024-09-15T03:04:55.777801+0000","last_fullsized":"2024-09-15T03:04:55.777801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:50:12.599469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778159+0000","last_change":"2024-09-15T03:04:55.777104+0000","last_active":"2024-09-15T03:04:55.778159+0000","last_peered":"2024-09-15T03:04:55.778159+0000","last_clean":"2024-09-15T03:04:55.778159+0000","last_became_active":"2024-09-15T03:04:55.774292+0000","last_became_peered":"2024-09-15T03:04:55.774292+0000","last_unstale":"2024-09-15T03:04:55.778159+0000","last_undegraded":"2024-09-15T03:04:55.778159+0000","last_fullsized":"2024-09-15T03:04:55.778159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:18.102228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":35104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":35104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":39003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":39003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:05:00.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:05:00.822 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:00.825 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:05:00.842 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-15T03:05:00.622068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189936,"kb_used_data":496,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297424,"statfs":{"total":386547056640,"available":385328562176,"internally_reserved":0,"allocated":507904,"data_stored":148214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.753985+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:05:05.130594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.777808+0000","last_change":"2024-09-15T03:04:55.775319+0000","last_active":"2024-09-15T03:04:55.777808+0000","last_peered":"2024-09-15T03:04:55.777808+0000","last_clean":"2024-09-15T03:04:55.777808+0000","last_became_active":"2024-09-15T03:04:55.774882+0000","last_became_peered":"2024-09-15T03:04:55.774882+0000","last_unstale":"2024-09-15T03:04:55.777808+0000","last_undegraded":"2024-09-15T03:04:55.777808+0000","last_fullsized":"2024-09-15T03:04:55.777808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:26:48.042045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.769265+0000","last_change":"2024-09-15T03:04:55.769265+0000","last_active":"2024-09-15T03:04:55.769265+0000","last_peered":"2024-09-15T03:04:55.769265+0000","last_clean":"2024-09-15T03:04:55.769265+0000","last_became_active":"2024-09-15T03:04:55.768372+0000","last_became_peered":"2024-09-15T03:04:55.768372+0000","last_unstale":"2024-09-15T03:04:55.769265+0000","last_undegraded":"2024-09-15T03:04:55.769265+0000","last_fullsized":"2024-09-15T03:04:55.769265+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:51.682552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778410+0000","last_change":"2024-09-15T03:04:55.777392+0000","last_active":"2024-09-15T03:04:55.778410+0000","last_peered":"2024-09-15T03:04:55.778410+0000","last_clean":"2024-09-15T03:04:55.778410+0000","last_became_active":"2024-09-15T03:04:55.767816+0000","last_became_peered":"2024-09-15T03:04:55.767816+0000","last_unstale":"2024-09-15T03:04:55.778410+0000","last_undegraded":"2024-09-15T03:04:55.778410+0000","last_fullsized":"2024-09-15T03:04:55.778410+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:49:25.460875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.753047+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:36:45.108947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.752703+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:22:44.914555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.771342+0000","last_change":"2024-09-15T03:04:55.770349+0000","last_active":"2024-09-15T03:04:55.771342+0000","last_peered":"2024-09-15T03:04:55.771342+0000","last_clean":"2024-09-15T03:04:55.771342+0000","last_became_active":"2024-09-15T03:04:55.769687+0000","last_became_peered":"2024-09-15T03:04:55.769687+0000","last_unstale":"2024-09-15T03:04:55.771342+0000","last_undegraded":"2024-09-15T03:04:55.771342+0000","last_fullsized":"2024-09-15T03:04:55.771342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:39:39.315986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778132+0000","last_change":"2024-09-15T03:04:55.775593+0000","last_active":"2024-09-15T03:04:55.778132+0000","last_peered":"2024-09-15T03:04:55.778132+0000","last_clean":"2024-09-15T03:04:55.778132+0000","last_became_active":"2024-09-15T03:04:55.774855+0000","last_became_peered":"2024-09-15T03:04:55.774855+0000","last_unstale":"2024-09-15T03:04:55.778132+0000","last_undegraded":"2024-09-15T03:04:55.778132+0000","last_fullsized":"2024-09-15T03:04:55.778132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:14:02.439822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.752135+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:07:22.230150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.763931+0000","last_change":"2024-09-15T03:04:55.763931+0000","last_active":"2024-09-15T03:04:55.763931+0000","last_peered":"2024-09-15T03:04:55.763931+0000","last_clean":"2024-09-15T03:04:55.763931+0000","last_became_active":"2024-09-15T03:04:55.763333+0000","last_became_peered":"2024-09-15T03:04:55.763333+0000","last_unstale":"2024-09-15T03:04:55.763931+0000","last_undegraded":"2024-09-15T03:04:55.763931+0000","last_fullsized":"2024-09-15T03:04:55.763931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:51:05.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T03:04:53.732736+0000","last_change":"2024-09-15T03:04:54.751529+0000","last_active":"2024-09-15T03:04:53.732736+0000","last_peered":"2024-09-15T03:04:53.732736+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T03:04:53.732736+0000","last_undegraded":"2024-09-15T03:04:53.732736+0000","last_fullsized":"2024-09-15T03:04:53.732736+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:36:59.232182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778371+0000","last_change":"2024-09-15T03:04:55.777926+0000","last_active":"2024-09-15T03:04:55.778371+0000","last_peered":"2024-09-15T03:04:55.778371+0000","last_clean":"2024-09-15T03:04:55.778371+0000","last_became_active":"2024-09-15T03:04:55.766750+0000","last_became_peered":"2024-09-15T03:04:55.766750+0000","last_unstale":"2024-09-15T03:04:55.778371+0000","last_undegraded":"2024-09-15T03:04:55.778371+0000","last_fullsized":"2024-09-15T03:04:55.778371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:47:19.790201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778389+0000","last_change":"2024-09-15T03:04:55.777093+0000","last_active":"2024-09-15T03:04:55.778389+0000","last_peered":"2024-09-15T03:04:55.778389+0000","last_clean":"2024-09-15T03:04:55.778389+0000","last_became_active":"2024-09-15T03:04:55.767200+0000","last_became_peered":"2024-09-15T03:04:55.767200+0000","last_unstale":"2024-09-15T03:04:55.778389+0000","last_undegraded":"2024-09-15T03:04:55.778389+0000","last_fullsized":"2024-09-15T03:04:55.778389+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:38.492407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.780670+0000","last_change":"2024-09-15T03:04:55.778257+0000","last_active":"2024-09-15T03:04:55.780670+0000","last_peered":"2024-09-15T03:04:55.780670+0000","last_clean":"2024-09-15T03:04:55.780670+0000","last_became_active":"2024-09-15T03:04:55.775400+0000","last_became_peered":"2024-09-15T03:04:55.775400+0000","last_unstale":"2024-09-15T03:04:55.780670+0000","last_undegraded":"2024-09-15T03:04:55.780670+0000","last_fullsized":"2024-09-15T03:04:55.780670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:02:00.435436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.777801+0000","last_change":"2024-09-15T03:04:55.774534+0000","last_active":"2024-09-15T03:04:55.777801+0000","last_peered":"2024-09-15T03:04:55.777801+0000","last_clean":"2024-09-15T03:04:55.777801+0000","last_became_active":"2024-09-15T03:04:55.766007+0000","last_became_peered":"2024-09-15T03:04:55.766007+0000","last_unstale":"2024-09-15T03:04:55.777801+0000","last_undegraded":"2024-09-15T03:04:55.777801+0000","last_fullsized":"2024-09-15T03:04:55.777801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:50:12.599469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778159+0000","last_change":"2024-09-15T03:04:55.777104+0000","last_active":"2024-09-15T03:04:55.778159+0000","last_peered":"2024-09-15T03:04:55.778159+0000","last_clean":"2024-09-15T03:04:55.778159+0000","last_became_active":"2024-09-15T03:04:55.774292+0000","last_became_peered":"2024-09-15T03:04:55.774292+0000","last_unstale":"2024-09-15T03:04:55.778159+0000","last_undegraded":"2024-09-15T03:04:55.778159+0000","last_fullsized":"2024-09-15T03:04:55.778159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:18.102228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":35104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":122880,"data_stored":35104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":39003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":39003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:05:03.844 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:05:04.148 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:04.152 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:05:04.162 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-15T03:05:02.622664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1190000,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297360,"statfs":{"total":386547056640,"available":385328496640,"internally_reserved":0,"allocated":573440,"data_stored":187864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.771241+0000","last_change":"2024-09-15T03:04:55.764708+0000","last_active":"2024-09-15T03:04:55.771241+0000","last_peered":"2024-09-15T03:04:55.771241+0000","last_clean":"2024-09-15T03:04:55.771241+0000","last_became_active":"2024-09-15T03:04:55.764222+0000","last_became_peered":"2024-09-15T03:04:55.764222+0000","last_unstale":"2024-09-15T03:04:55.771241+0000","last_undegraded":"2024-09-15T03:04:55.771241+0000","last_fullsized":"2024-09-15T03:04:55.771241+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:16:01.401691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.777808+0000","last_change":"2024-09-15T03:04:55.775319+0000","last_active":"2024-09-15T03:04:55.777808+0000","last_peered":"2024-09-15T03:04:55.777808+0000","last_clean":"2024-09-15T03:04:55.777808+0000","last_became_active":"2024-09-15T03:04:55.774882+0000","last_became_peered":"2024-09-15T03:04:55.774882+0000","last_unstale":"2024-09-15T03:04:55.777808+0000","last_undegraded":"2024-09-15T03:04:55.777808+0000","last_fullsized":"2024-09-15T03:04:55.777808+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:26:48.042045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.774697+0000","last_change":"2024-09-15T03:04:55.769265+0000","last_active":"2024-09-15T03:04:55.774697+0000","last_peered":"2024-09-15T03:04:55.774697+0000","last_clean":"2024-09-15T03:04:55.774697+0000","last_became_active":"2024-09-15T03:04:55.768372+0000","last_became_peered":"2024-09-15T03:04:55.768372+0000","last_unstale":"2024-09-15T03:04:55.774697+0000","last_undegraded":"2024-09-15T03:04:55.774697+0000","last_fullsized":"2024-09-15T03:04:55.774697+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:51.682552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778410+0000","last_change":"2024-09-15T03:04:55.777392+0000","last_active":"2024-09-15T03:04:55.778410+0000","last_peered":"2024-09-15T03:04:55.778410+0000","last_clean":"2024-09-15T03:04:55.778410+0000","last_became_active":"2024-09-15T03:04:55.767816+0000","last_became_peered":"2024-09-15T03:04:55.767816+0000","last_unstale":"2024-09-15T03:04:55.778410+0000","last_undegraded":"2024-09-15T03:04:55.778410+0000","last_fullsized":"2024-09-15T03:04:55.778410+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:49:25.460875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.771793+0000","last_change":"2024-09-15T03:04:55.767899+0000","last_active":"2024-09-15T03:04:55.771793+0000","last_peered":"2024-09-15T03:04:55.771793+0000","last_clean":"2024-09-15T03:04:55.771793+0000","last_became_active":"2024-09-15T03:04:55.766951+0000","last_became_peered":"2024-09-15T03:04:55.766951+0000","last_unstale":"2024-09-15T03:04:55.771793+0000","last_undegraded":"2024-09-15T03:04:55.771793+0000","last_fullsized":"2024-09-15T03:04:55.771793+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:44:43.008913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.772227+0000","last_change":"2024-09-15T03:04:55.771091+0000","last_active":"2024-09-15T03:04:55.772227+0000","last_peered":"2024-09-15T03:04:55.772227+0000","last_clean":"2024-09-15T03:04:55.772227+0000","last_became_active":"2024-09-15T03:04:55.767439+0000","last_became_peered":"2024-09-15T03:04:55.767439+0000","last_unstale":"2024-09-15T03:04:55.772227+0000","last_undegraded":"2024-09-15T03:04:55.772227+0000","last_fullsized":"2024-09-15T03:04:55.772227+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:58:50.162650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.771342+0000","last_change":"2024-09-15T03:04:55.770349+0000","last_active":"2024-09-15T03:04:55.771342+0000","last_peered":"2024-09-15T03:04:55.771342+0000","last_clean":"2024-09-15T03:04:55.771342+0000","last_became_active":"2024-09-15T03:04:55.769687+0000","last_became_peered":"2024-09-15T03:04:55.769687+0000","last_unstale":"2024-09-15T03:04:55.771342+0000","last_undegraded":"2024-09-15T03:04:55.771342+0000","last_fullsized":"2024-09-15T03:04:55.771342+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:39:39.315986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778132+0000","last_change":"2024-09-15T03:04:55.775593+0000","last_active":"2024-09-15T03:04:55.778132+0000","last_peered":"2024-09-15T03:04:55.778132+0000","last_clean":"2024-09-15T03:04:55.778132+0000","last_became_active":"2024-09-15T03:04:55.774855+0000","last_became_peered":"2024-09-15T03:04:55.774855+0000","last_unstale":"2024-09-15T03:04:55.778132+0000","last_undegraded":"2024-09-15T03:04:55.778132+0000","last_fullsized":"2024-09-15T03:04:55.778132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:14:02.439822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"16'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:04:56.772455+0000","last_change":"2024-09-15T03:04:55.764765+0000","last_active":"2024-09-15T03:04:56.772455+0000","last_peered":"2024-09-15T03:04:56.772455+0000","last_clean":"2024-09-15T03:04:56.772455+0000","last_became_active":"2024-09-15T03:04:55.764295+0000","last_became_peered":"2024-09-15T03:04:55.764295+0000","last_unstale":"2024-09-15T03:04:56.772455+0000","last_undegraded":"2024-09-15T03:04:56.772455+0000","last_fullsized":"2024-09-15T03:04:56.772455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:23:22.071382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.769371+0000","last_change":"2024-09-15T03:04:55.763931+0000","last_active":"2024-09-15T03:04:55.769371+0000","last_peered":"2024-09-15T03:04:55.769371+0000","last_clean":"2024-09-15T03:04:55.769371+0000","last_became_active":"2024-09-15T03:04:55.763333+0000","last_became_peered":"2024-09-15T03:04:55.763333+0000","last_unstale":"2024-09-15T03:04:55.769371+0000","last_undegraded":"2024-09-15T03:04:55.769371+0000","last_fullsized":"2024-09-15T03:04:55.769371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:51:05.382278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.771810+0000","last_change":"2024-09-15T03:04:55.765507+0000","last_active":"2024-09-15T03:04:55.771810+0000","last_peered":"2024-09-15T03:04:55.771810+0000","last_clean":"2024-09-15T03:04:55.771810+0000","last_became_active":"2024-09-15T03:04:55.765075+0000","last_became_peered":"2024-09-15T03:04:55.765075+0000","last_unstale":"2024-09-15T03:04:55.771810+0000","last_undegraded":"2024-09-15T03:04:55.771810+0000","last_fullsized":"2024-09-15T03:04:55.771810+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:44:58.673716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778371+0000","last_change":"2024-09-15T03:04:55.777926+0000","last_active":"2024-09-15T03:04:55.778371+0000","last_peered":"2024-09-15T03:04:55.778371+0000","last_clean":"2024-09-15T03:04:55.778371+0000","last_became_active":"2024-09-15T03:04:55.766750+0000","last_became_peered":"2024-09-15T03:04:55.766750+0000","last_unstale":"2024-09-15T03:04:55.778371+0000","last_undegraded":"2024-09-15T03:04:55.778371+0000","last_fullsized":"2024-09-15T03:04:55.778371+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:47:19.790201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778389+0000","last_change":"2024-09-15T03:04:55.777093+0000","last_active":"2024-09-15T03:04:55.778389+0000","last_peered":"2024-09-15T03:04:55.778389+0000","last_clean":"2024-09-15T03:04:55.778389+0000","last_became_active":"2024-09-15T03:04:55.767200+0000","last_became_peered":"2024-09-15T03:04:55.767200+0000","last_unstale":"2024-09-15T03:04:55.778389+0000","last_undegraded":"2024-09-15T03:04:55.778389+0000","last_fullsized":"2024-09-15T03:04:55.778389+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:14:38.492407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.780670+0000","last_change":"2024-09-15T03:04:55.778257+0000","last_active":"2024-09-15T03:04:55.780670+0000","last_peered":"2024-09-15T03:04:55.780670+0000","last_clean":"2024-09-15T03:04:55.780670+0000","last_became_active":"2024-09-15T03:04:55.775400+0000","last_became_peered":"2024-09-15T03:04:55.775400+0000","last_unstale":"2024-09-15T03:04:55.780670+0000","last_undegraded":"2024-09-15T03:04:55.780670+0000","last_fullsized":"2024-09-15T03:04:55.780670+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:02:00.435436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.777801+0000","last_change":"2024-09-15T03:04:55.774534+0000","last_active":"2024-09-15T03:04:55.777801+0000","last_peered":"2024-09-15T03:04:55.777801+0000","last_clean":"2024-09-15T03:04:55.777801+0000","last_became_active":"2024-09-15T03:04:55.766007+0000","last_became_peered":"2024-09-15T03:04:55.766007+0000","last_unstale":"2024-09-15T03:04:55.777801+0000","last_undegraded":"2024-09-15T03:04:55.777801+0000","last_fullsized":"2024-09-15T03:04:55.777801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:50:12.599469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":14,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:04:55.778159+0000","last_change":"2024-09-15T03:04:55.777104+0000","last_active":"2024-09-15T03:04:55.778159+0000","last_peered":"2024-09-15T03:04:55.778159+0000","last_clean":"2024-09-15T03:04:55.778159+0000","last_became_active":"2024-09-15T03:04:55.774292+0000","last_became_peered":"2024-09-15T03:04:55.774292+0000","last_unstale":"2024-09-15T03:04:55.778159+0000","last_undegraded":"2024-09-15T03:04:55.778159+0000","last_fullsized":"2024-09-15T03:04:55.778159+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:13:18.102228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":143360,"data_stored":46966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:05:04.163 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-15T03:05:04.163 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-15T03:05:04.164 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-15T03:05:04.218 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-15T03:05:04.218 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-15T03:05:04.264 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2024-09-15T03:05:04.264 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2024-09-15T03:05:04.314 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-15T03:05:04.314 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-15T03:05:04.364 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-15T03:05:04.364 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-15T03:05:04.413 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2024-09-15T03:05:04.413 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2024-09-15T03:05:04.463 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-15T03:05:04.463 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-15T03:05:04.513 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-15T03:05:04.514 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-15T03:05:04.564 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2024-09-15T03:05:04.564 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2024-09-15T03:05:04.614 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-15T03:05:04.614 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-15T03:05:04.667 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-15T03:05:04.667 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-15T03:05:04.718 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2024-09-15T03:05:04.718 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2024-09-15T03:05:04.763 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-15T03:05:04.764 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-15T03:05:04.815 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-15T03:05:04.815 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-15T03:05:04.865 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2024-09-15T03:05:04.865 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2024-09-15T03:05:04.913 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-15T03:05:04.913 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-15T03:05:04.964 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-15T03:05:04.964 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-15T03:05:05.014 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2024-09-15T03:05:05.015 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2024-09-15T03:05:05.061 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-15T03:05:05.061 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-15T03:05:05.114 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-15T03:05:05.114 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-15T03:05:05.166 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2024-09-15T03:05:05.166 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2024-09-15T03:05:05.212 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-15T03:05:05.212 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-15T03:05:05.261 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-15T03:05:05.262 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-15T03:05:05.314 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2024-09-15T03:05:05.314 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2024-09-15T03:05:05.361 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-15T03:05:05.361 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-15T03:05:05.408 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-15T03:05:05.408 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-15T03:05:05.459 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2024-09-15T03:05:05.459 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2024-09-15T03:05:05.511 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-15T03:05:05.658 INFO:teuthology.orchestra.run.smithi060.stdout:{} 2024-09-15T03:05:05.660 INFO:teuthology.orchestra.run.smithi060.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-15T03:05:05.673 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T03:05:05.673 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:05:05.758 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T03:05:05.758 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-15T03:05:06.052 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-15T03:05:06.045+0000 7f50cc5e7700 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-15T03:05:06.056 INFO:teuthology.orchestra.run.smithi060.stderr:osd.0 is already down. 2024-09-15T03:05:06.067 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T03:05:06.067 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:05:06.140 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T03:05:06.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-15T03:05:06.440 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-15T03:05:06.433+0000 7f50cc5e7700 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-15T03:05:06.872 INFO:teuthology.orchestra.run.smithi060.stderr:marked down osd.3. 2024-09-15T03:05:06.883 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2024-09-15T03:05:06.883 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2024-09-15T03:05:07.884 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-15T03:05:07.884 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-15T03:05:07.929 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-15T03:05:07.929 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-15T03:05:07.978 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2024-09-15T03:05:07.978 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2024-09-15T03:05:08.026 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-15T03:05:08.026 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-15T03:05:08.077 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-15T03:05:08.078 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-15T03:05:08.126 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2024-09-15T03:05:08.127 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2024-09-15T03:05:08.179 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-15T03:05:08.179 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-15T03:05:08.234 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-15T03:05:08.234 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-15T03:05:08.286 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2024-09-15T03:05:08.287 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2024-09-15T03:05:08.339 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-15T03:05:08.339 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-15T03:05:08.392 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-15T03:05:08.392 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-15T03:05:08.444 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2024-09-15T03:05:08.444 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2024-09-15T03:05:08.494 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-15T03:05:08.494 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-15T03:05:08.547 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-15T03:05:08.547 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-15T03:05:08.598 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2024-09-15T03:05:08.598 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2024-09-15T03:05:08.648 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-15T03:05:08.649 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-15T03:05:08.698 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-15T03:05:08.698 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-15T03:05:08.745 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2024-09-15T03:05:08.745 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2024-09-15T03:05:08.793 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-15T03:05:08.793 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-15T03:05:08.845 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-15T03:05:08.845 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-15T03:05:08.895 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2024-09-15T03:05:08.895 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2024-09-15T03:05:08.949 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-15T03:05:08.949 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-15T03:05:09.004 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-15T03:05:09.004 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-15T03:05:09.057 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2024-09-15T03:05:09.058 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2024-09-15T03:05:09.108 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-15T03:05:09.108 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-15T03:05:09.160 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-15T03:05:09.160 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-15T03:05:09.214 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T03:05:09.214 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:05:09.278 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T03:05:09.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-15T03:05:09.594 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-15T03:05:09.585+0000 7f50cc5e7700 -1 mon.a@0(leader).osd e19 definitely_dead 0 2024-09-15T03:05:09.890 INFO:teuthology.orchestra.run.smithi060.stderr:marked down osd.1. 2024-09-15T03:05:09.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-15T03:05:10.898 INFO:teuthology.orchestra.run.smithi060.stderr:marked osd lost in epoch 20 2024-09-15T03:05:10.909 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T03:05:10.910 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T03:05:10.911 INFO:tasks.ceph.osd.0:Started 2024-09-15T03:05:10.912 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:05:11.006 INFO:teuthology.orchestra.run.smithi060.stderr:no valid command found; 10 closest matches: 2024-09-15T03:05:11.006 INFO:teuthology.orchestra.run.smithi060.stderr:0 2024-09-15T03:05:11.006 INFO:teuthology.orchestra.run.smithi060.stderr:1 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:2 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:abort 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:assert 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:config diff 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:config diff get 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:config get 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:config help [] 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:config set ... 2024-09-15T03:05:11.007 INFO:teuthology.orchestra.run.smithi060.stderr:admin_socket: invalid command 2024-09-15T03:05:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T03:05:11.009 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-15T03:05:11.512 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:05:11.505+0000 7fac50349540 -1 Falling back to public interface 2024-09-15T03:05:14.133 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:05:14.125+0000 7fac50349540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:05:14.921 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:05:14.913+0000 7fac50349540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:05:15.255 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:05:15.249+0000 7fac50349540 -1 osd.0 16 log_to_monitors true 2024-09-15T03:05:15.965 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-09-15T03:05:15.957+0000 7fac46890700 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:05:16.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:16.131 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2024-09-15T03:05:16.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:05:16.419 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:16.419 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":22,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:05:15.943813+0000","last_up_change":"2024-09-15T03:05:09.870227+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:04:52.771663+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":22281},{"type":"v1","addr":"172.21.15.60:6801","nonce":22281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":22281},{"type":"v1","addr":"172.21.15.60:6803","nonce":22281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":22281},{"type":"v1","addr":"172.21.15.60:6807","nonce":22281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":22281},{"type":"v1","addr":"172.21.15.60:6805","nonce":22281}]},"public_addr":"172.21.15.60:6801/22281","cluster_addr":"172.21.15.60:6803/22281","heartbeat_back_addr":"172.21.15.60:6807/22281","heartbeat_front_addr":"172.21.15.60:6805/22281","state":["exists"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":22282},{"type":"v1","addr":"172.21.15.60:6809","nonce":22282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":22282},{"type":"v1","addr":"172.21.15.60:6811","nonce":22282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":22282},{"type":"v1","addr":"172.21.15.60:6815","nonce":22282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":22282},{"type":"v1","addr":"172.21.15.60:6813","nonce":22282}]},"public_addr":"172.21.15.60:6809/22282","cluster_addr":"172.21.15.60:6811/22282","heartbeat_back_addr":"172.21.15.60:6815/22282","heartbeat_front_addr":"172.21.15.60:6813/22282","state":["exists"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":22284},{"type":"v1","addr":"172.21.15.60:6817","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":22284},{"type":"v1","addr":"172.21.15.60:6819","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":22284},{"type":"v1","addr":"172.21.15.60:6823","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":22284},{"type":"v1","addr":"172.21.15.60:6821","nonce":22284}]},"public_addr":"172.21.15.60:6817/22284","cluster_addr":"172.21.15.60:6819/22284","heartbeat_back_addr":"172.21.15.60:6823/22284","heartbeat_front_addr":"172.21.15.60:6821/22284","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:05:19.433 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:05:19.729 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:19.730 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":24,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:05:17.965649+0000","last_up_change":"2024-09-15T03:05:16.952480+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:04:52.771663+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":28132},{"type":"v1","addr":"172.21.15.60:6801","nonce":28132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":28132},{"type":"v1","addr":"172.21.15.60:6803","nonce":28132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":28132},{"type":"v1","addr":"172.21.15.60:6807","nonce":28132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":28132},{"type":"v1","addr":"172.21.15.60:6805","nonce":28132}]},"public_addr":"172.21.15.60:6801/28132","cluster_addr":"172.21.15.60:6803/28132","heartbeat_back_addr":"172.21.15.60:6807/28132","heartbeat_front_addr":"172.21.15.60:6805/28132","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":22282},{"type":"v1","addr":"172.21.15.60:6809","nonce":22282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":22282},{"type":"v1","addr":"172.21.15.60:6811","nonce":22282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":22282},{"type":"v1","addr":"172.21.15.60:6815","nonce":22282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":22282},{"type":"v1","addr":"172.21.15.60:6813","nonce":22282}]},"public_addr":"172.21.15.60:6809/22282","cluster_addr":"172.21.15.60:6811/22282","heartbeat_back_addr":"172.21.15.60:6815/22282","heartbeat_front_addr":"172.21.15.60:6813/22282","state":["exists"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":22284},{"type":"v1","addr":"172.21.15.60:6817","nonce":22284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":22284},{"type":"v1","addr":"172.21.15.60:6819","nonce":22284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":22284},{"type":"v1","addr":"172.21.15.60:6823","nonce":22284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":22284},{"type":"v1","addr":"172.21.15.60:6821","nonce":22284}]},"public_addr":"172.21.15.60:6817/22284","cluster_addr":"172.21.15.60:6819/22284","heartbeat_back_addr":"172.21.15.60:6823/22284","heartbeat_front_addr":"172.21.15.60:6821/22284","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:05:19.744 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2024-09-15T03:05:19.744 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T03:05:19.744 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T03:05:19.746 INFO:tasks.ceph.osd.3:Started 2024-09-15T03:05:19.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:05:19.844 INFO:teuthology.orchestra.run.smithi060.stderr:no valid command found; 10 closest matches: 2024-09-15T03:05:19.844 INFO:teuthology.orchestra.run.smithi060.stderr:0 2024-09-15T03:05:19.844 INFO:teuthology.orchestra.run.smithi060.stderr:1 2024-09-15T03:05:19.844 INFO:teuthology.orchestra.run.smithi060.stderr:2 2024-09-15T03:05:19.845 INFO:teuthology.orchestra.run.smithi060.stderr:abort 2024-09-15T03:05:19.845 INFO:teuthology.orchestra.run.smithi060.stderr:assert 2024-09-15T03:05:19.845 INFO:teuthology.orchestra.run.smithi060.stderr:config diff 2024-09-15T03:05:19.845 INFO:teuthology.orchestra.run.smithi060.stderr:config diff get 2024-09-15T03:05:19.845 INFO:teuthology.orchestra.run.smithi060.stderr:config get 2024-09-15T03:05:19.845 INFO:teuthology.orchestra.run.smithi060.stderr:config help [] 2024-09-15T03:05:19.846 INFO:teuthology.orchestra.run.smithi060.stderr:config set ... 2024-09-15T03:05:19.846 INFO:teuthology.orchestra.run.smithi060.stderr:admin_socket: invalid command 2024-09-15T03:05:19.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T03:05:19.847 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-15T03:05:20.360 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:05:20.353+0000 7fea811c0540 -1 Falling back to public interface 2024-09-15T03:05:22.988 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-15T03:05:22.981+0000 7f50cedec700 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-15T03:05:23.005 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:05:22.997+0000 7fea811c0540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:05:23.791 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:05:23.785+0000 7fea811c0540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:05:24.123 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:05:24.117+0000 7fea811c0540 -1 osd.3 17 log_to_monitors true 2024-09-15T03:05:24.848 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:05:24.956 INFO:teuthology.orchestra.run.smithi060.stderr:no valid command found; 10 closest matches: 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:0 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:1 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:2 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:abort 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:assert 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:bluefs debug_inject_read_zeros 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:bluefs files list 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:bluefs stats 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:bluestore allocator dump block 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:bluestore allocator fragmentation block 2024-09-15T03:05:24.957 INFO:teuthology.orchestra.run.smithi060.stderr:admin_socket: invalid command 2024-09-15T03:05:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T03:05:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-15T03:05:25.063 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-09-15T03:05:25.057+0000 7fea78f0a700 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:05:29.086 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-15T03:05:29.077+0000 7f50cedec700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 10 pgs recovery_unfound (PG_DAMAGED) 2024-09-15T03:05:29.960 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T03:05:30.072 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:30.072 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-15T03:05:30.072 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-15T03:05:30.072 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:30.081 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2024-09-15T03:05:30.081 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:05:30.368 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:30.369 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":27,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:05:27.051386+0000","last_up_change":"2024-09-15T03:05:26.048611+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:04:52.771663+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":26,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":28132},{"type":"v1","addr":"172.21.15.60:6801","nonce":28132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":28132},{"type":"v1","addr":"172.21.15.60:6803","nonce":28132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":28132},{"type":"v1","addr":"172.21.15.60:6807","nonce":28132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":28132},{"type":"v1","addr":"172.21.15.60:6805","nonce":28132}]},"public_addr":"172.21.15.60:6801/28132","cluster_addr":"172.21.15.60:6803/28132","heartbeat_back_addr":"172.21.15.60:6807/28132","heartbeat_front_addr":"172.21.15.60:6805/28132","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":22282},{"type":"v1","addr":"172.21.15.60:6809","nonce":22282}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":22282},{"type":"v1","addr":"172.21.15.60:6811","nonce":22282}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":22282},{"type":"v1","addr":"172.21.15.60:6815","nonce":22282}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":22282},{"type":"v1","addr":"172.21.15.60:6813","nonce":22282}]},"public_addr":"172.21.15.60:6809/22282","cluster_addr":"172.21.15.60:6811/22282","heartbeat_back_addr":"172.21.15.60:6815/22282","heartbeat_front_addr":"172.21.15.60:6813/22282","state":["exists"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":26,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":29026},{"type":"v1","addr":"172.21.15.60:6809","nonce":29026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":29026},{"type":"v1","addr":"172.21.15.60:6811","nonce":29026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":29026},{"type":"v1","addr":"172.21.15.60:6815","nonce":29026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":29026},{"type":"v1","addr":"172.21.15.60:6813","nonce":29026}]},"public_addr":"172.21.15.60:6809/29026","cluster_addr":"172.21.15.60:6811/29026","heartbeat_back_addr":"172.21.15.60:6815/29026","heartbeat_front_addr":"172.21.15.60:6813/29026","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:05:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2024-09-15T03:05:30.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:05:30.383 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:05:30.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:05:30.522 INFO:teuthology.orchestra.run.smithi060.stdout:111669149699 2024-09-15T03:05:30.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:05:30.523 INFO:teuthology.orchestra.run.smithi060.stdout:98784247812 2024-09-15T03:05:30.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:05:30.528 INFO:teuthology.orchestra.run.smithi060.stdout:38654705675 2024-09-15T03:05:30.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:05:30.790 INFO:teuthology.orchestra.run.smithi060.stdout:98784247811 2024-09-15T03:05:30.798 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-09-15T03:05:30.804 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247812 got 98784247811 for osd.0 2024-09-15T03:05:30.811 INFO:teuthology.orchestra.run.smithi060.stdout:38654705674 2024-09-15T03:05:30.812 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149699 got 1 for osd.3 2024-09-15T03:05:30.824 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705675 got 38654705674 for osd.2 2024-09-15T03:05:31.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:05:31.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:05:31.825 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:05:32.102 INFO:teuthology.orchestra.run.smithi060.stdout:111669149699 2024-09-15T03:05:32.115 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149699 got 111669149699 for osd.3 2024-09-15T03:05:32.116 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:32.120 INFO:teuthology.orchestra.run.smithi060.stdout:98784247812 2024-09-15T03:05:32.132 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247812 got 98784247812 for osd.0 2024-09-15T03:05:32.133 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:32.147 INFO:teuthology.orchestra.run.smithi060.stdout:38654705675 2024-09-15T03:05:32.163 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705675 got 38654705675 for osd.2 2024-09-15T03:05:32.164 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:32.164 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2024-09-15T03:05:32.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:05:32.440 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:32.443 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:05:32.457 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-15T03:05:30.633066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1191448,"kb_used_data":1044,"kb_used_omap":12,"kb_used_meta":1190323,"kb_avail":376295912,"statfs":{"total":386547056640,"available":385327013888,"internally_reserved":0,"allocated":1069056,"data_stored":540758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1218891570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002547"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-15T03:05:27.078349+0000","last_change":"2024-09-15T03:05:27.078107+0000","last_active":"2024-09-15T03:05:27.078349+0000","last_peered":"2024-09-15T03:05:27.078349+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.078107+0000","last_became_peered":"2024-09-15T03:05:27.078107+0000","last_unstale":"2024-09-15T03:05:27.078349+0000","last_undegraded":"2024-09-15T03:05:27.078349+0000","last_fullsized":"2024-09-15T03:05:27.059184+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:26.729372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.084497+0000","last_change":"2024-09-15T03:05:27.084884+0000","last_active":"2024-09-15T03:05:27.084497+0000","last_peered":"2024-09-15T03:05:27.084497+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:05:27.076040+0000","last_became_peered":"2024-09-15T03:05:27.076040+0000","last_unstale":"2024-09-15T03:05:27.084497+0000","last_undegraded":"2024-09-15T03:05:27.063502+0000","last_fullsized":"2024-09-15T03:05:27.062931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:49:47.581517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":61,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.075646+0000","last_change":"2024-09-15T03:05:27.076017+0000","last_active":"2024-09-15T03:05:27.075646+0000","last_peered":"2024-09-15T03:05:27.075646+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:05:27.073076+0000","last_became_peered":"2024-09-15T03:05:27.073076+0000","last_unstale":"2024-09-15T03:05:27.075646+0000","last_undegraded":"2024-09-15T03:05:27.062455+0000","last_fullsized":"2024-09-15T03:05:27.061642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:12.173681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.089693+0000","last_change":"2024-09-15T03:05:27.090005+0000","last_active":"2024-09-15T03:05:27.089693+0000","last_peered":"2024-09-15T03:05:27.089693+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:05:27.077226+0000","last_became_peered":"2024-09-15T03:05:27.077226+0000","last_unstale":"2024-09-15T03:05:27.089693+0000","last_undegraded":"2024-09-15T03:05:27.064312+0000","last_fullsized":"2024-09-15T03:05:27.063909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:26.641641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":33,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.085259+0000","last_change":"2024-09-15T03:05:27.085785+0000","last_active":"2024-09-15T03:05:27.085259+0000","last_peered":"2024-09-15T03:05:27.085259+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:05:27.070138+0000","last_became_peered":"2024-09-15T03:05:27.070138+0000","last_unstale":"2024-09-15T03:05:27.085259+0000","last_undegraded":"2024-09-15T03:05:27.057975+0000","last_fullsized":"2024-09-15T03:05:27.057143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:39:19.992759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-15T03:05:27.069627+0000","last_change":"2024-09-15T03:05:27.069510+0000","last_active":"2024-09-15T03:05:27.069627+0000","last_peered":"2024-09-15T03:05:27.069627+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.069510+0000","last_became_peered":"2024-09-15T03:05:27.069510+0000","last_unstale":"2024-09-15T03:05:27.069627+0000","last_undegraded":"2024-09-15T03:05:27.069627+0000","last_fullsized":"2024-09-15T03:05:27.057867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:56.697498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":46,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.081119+0000","last_change":"2024-09-15T03:05:27.081660+0000","last_active":"2024-09-15T03:05:27.081119+0000","last_peered":"2024-09-15T03:05:27.081119+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:05:27.073004+0000","last_became_peered":"2024-09-15T03:05:27.073004+0000","last_unstale":"2024-09-15T03:05:27.081119+0000","last_undegraded":"2024-09-15T03:05:27.062946+0000","last_fullsized":"2024-09-15T03:05:27.061959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:02:15.775095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":45,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.087874+0000","last_change":"2024-09-15T03:05:27.088312+0000","last_active":"2024-09-15T03:05:27.087874+0000","last_peered":"2024-09-15T03:05:27.087874+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:05:27.076435+0000","last_became_peered":"2024-09-15T03:05:27.076435+0000","last_unstale":"2024-09-15T03:05:27.087874+0000","last_undegraded":"2024-09-15T03:05:27.064600+0000","last_fullsized":"2024-09-15T03:05:27.063343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:05:12.788710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":38,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.091460+0000","last_change":"2024-09-15T03:05:27.091808+0000","last_active":"2024-09-15T03:05:27.091460+0000","last_peered":"2024-09-15T03:05:27.091460+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:05:27.079178+0000","last_became_peered":"2024-09-15T03:05:27.079178+0000","last_unstale":"2024-09-15T03:05:27.091460+0000","last_undegraded":"2024-09-15T03:05:27.058589+0000","last_fullsized":"2024-09-15T03:05:27.057594+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:03:17.277311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":59,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.079377+0000","last_change":"2024-09-15T03:05:27.079875+0000","last_active":"2024-09-15T03:05:27.079377+0000","last_peered":"2024-09-15T03:05:27.079377+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:05:27.073080+0000","last_became_peered":"2024-09-15T03:05:27.073080+0000","last_unstale":"2024-09-15T03:05:27.079377+0000","last_undegraded":"2024-09-15T03:05:27.060756+0000","last_fullsized":"2024-09-15T03:05:27.059796+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:05:43.290430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":34,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.082099+0000","last_change":"2024-09-15T03:05:27.082524+0000","last_active":"2024-09-15T03:05:27.082099+0000","last_peered":"2024-09-15T03:05:27.082099+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:05:27.069721+0000","last_became_peered":"2024-09-15T03:05:27.069721+0000","last_unstale":"2024-09-15T03:05:27.082099+0000","last_undegraded":"2024-09-15T03:05:27.058374+0000","last_fullsized":"2024-09-15T03:05:27.057414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:25.934882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":19,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.088652+0000","last_change":"2024-09-15T03:05:27.088981+0000","last_active":"2024-09-15T03:05:27.088652+0000","last_peered":"2024-09-15T03:05:27.088652+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:05:27.070691+0000","last_became_peered":"2024-09-15T03:05:27.070691+0000","last_unstale":"2024-09-15T03:05:27.088652+0000","last_undegraded":"2024-09-15T03:05:27.064197+0000","last_fullsized":"2024-09-15T03:05:27.063869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:19:59.460163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":58,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.083506+0000","last_change":"2024-09-15T03:05:27.083985+0000","last_active":"2024-09-15T03:05:27.083506+0000","last_peered":"2024-09-15T03:05:27.083506+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:05:27.073296+0000","last_became_peered":"2024-09-15T03:05:27.073296+0000","last_unstale":"2024-09-15T03:05:27.083506+0000","last_undegraded":"2024-09-15T03:05:27.065069+0000","last_fullsized":"2024-09-15T03:05:27.064221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:22:11.122806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":31,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.090312+0000","last_change":"2024-09-15T03:05:27.090687+0000","last_active":"2024-09-15T03:05:27.090312+0000","last_peered":"2024-09-15T03:05:27.090312+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:05:27.071179+0000","last_became_peered":"2024-09-15T03:05:27.071179+0000","last_unstale":"2024-09-15T03:05:27.090312+0000","last_undegraded":"2024-09-15T03:05:27.059665+0000","last_fullsized":"2024-09-15T03:05:27.057178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:36:34.628592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":61,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.087212+0000","last_change":"2024-09-15T03:05:27.087543+0000","last_active":"2024-09-15T03:05:27.087212+0000","last_peered":"2024-09-15T03:05:27.087212+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:05:27.074164+0000","last_became_peered":"2024-09-15T03:05:27.074164+0000","last_unstale":"2024-09-15T03:05:27.087212+0000","last_undegraded":"2024-09-15T03:05:27.063361+0000","last_fullsized":"2024-09-15T03:05:27.062449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:45:22.140009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":45,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:27.086501+0000","last_change":"2024-09-15T03:05:27.086812+0000","last_active":"2024-09-15T03:05:27.086501+0000","last_peered":"2024-09-15T03:05:27.086501+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:05:27.076127+0000","last_became_peered":"2024-09-15T03:05:27.076127+0000","last_unstale":"2024-09-15T03:05:27.086501+0000","last_undegraded":"2024-09-15T03:05:27.063025+0000","last_fullsized":"2024-09-15T03:05:27.061985+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:21:26.414306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298140,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073700,"statfs":{"total":96636764160,"available":96331468800,"internally_reserved":0,"allocated":270336,"data_stored":126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297692,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074148,"statfs":{"total":96636764160,"available":96331927552,"internally_reserved":0,"allocated":339968,"data_stored":199525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]}]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":188416,"data_stored":87739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298076,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":94073764,"statfs":{"total":96636764160,"available":96331534336,"internally_reserved":0,"allocated":270336,"data_stored":126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:05:32.459 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:05:32.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:32.798 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":55,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726369526,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":1220042752,"bytes_avail":385327013888,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-15T03:04:36.947679+0000","services":{}},"progress_events":{}} 2024-09-15T03:05:32.812 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 55, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726369526, 'num_in_osds': 4, 'osd_in_since': 1726369479, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 195132, 'bytes_used': 1220042752, 'bytes_avail': 385327013888, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-15T03:04:36.947679+0000', 'services': {}}, 'progress_events': {}} 2024-09-15T03:05:32.812 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2024-09-15T03:05:32.812 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:05:32.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:05:32.813 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:05:32.943 INFO:teuthology.orchestra.run.smithi060.stdout:38654705677 2024-09-15T03:05:32.944 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:05:32.962 INFO:teuthology.orchestra.run.smithi060.stdout:111669149700 2024-09-15T03:05:32.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:05:32.967 INFO:teuthology.orchestra.run.smithi060.stdout:98784247814 2024-09-15T03:05:32.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:05:33.237 INFO:teuthology.orchestra.run.smithi060.stdout:38654705676 2024-09-15T03:05:33.251 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705677 got 38654705676 for osd.2 2024-09-15T03:05:33.264 INFO:teuthology.orchestra.run.smithi060.stdout:111669149699 2024-09-15T03:05:33.271 INFO:teuthology.orchestra.run.smithi060.stdout:98784247813 2024-09-15T03:05:33.284 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149699 for osd.3 2024-09-15T03:05:33.285 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-15T03:05:34.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:05:34.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:05:34.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:05:34.546 INFO:teuthology.orchestra.run.smithi060.stdout:38654705676 2024-09-15T03:05:34.566 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705677 got 38654705676 for osd.2 2024-09-15T03:05:34.572 INFO:teuthology.orchestra.run.smithi060.stdout:111669149699 2024-09-15T03:05:34.573 INFO:teuthology.orchestra.run.smithi060.stdout:98784247813 2024-09-15T03:05:34.585 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149699 for osd.3 2024-09-15T03:05:34.586 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-15T03:05:35.567 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:05:35.586 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:05:35.586 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:05:35.896 INFO:teuthology.orchestra.run.smithi060.stdout:98784247814 2024-09-15T03:05:35.914 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247814 for osd.0 2024-09-15T03:05:35.914 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:35.922 INFO:teuthology.orchestra.run.smithi060.stdout:111669149700 2024-09-15T03:05:35.932 INFO:teuthology.orchestra.run.smithi060.stdout:38654705677 2024-09-15T03:05:35.941 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149700 for osd.3 2024-09-15T03:05:35.941 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:35.947 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705677 got 38654705677 for osd.2 2024-09-15T03:05:35.947 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:35.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:05:36.286 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:36.286 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":59,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726369526,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":1220042752,"bytes_avail":385327013888,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-15T03:04:36.947679+0000","services":{}},"progress_events":{}} 2024-09-15T03:05:36.299 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 59, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726369526, 'num_in_osds': 4, 'osd_in_since': 1726369479, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 195132, 'bytes_used': 1220042752, 'bytes_avail': 385327013888, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-15T03:04:36.947679+0000', 'services': {}}, 'progress_events': {}} 2024-09-15T03:05:36.300 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2024-09-15T03:05:36.300 DEBUG:teuthology.orchestra.run.smithi060:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2024-09-15T03:05:36.347 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stderr:INFO: op_size has been rounded to 8192 2024-09-15T03:05:36.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:hints = 1 2024-09-15T03:05:36.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2024-09-15T03:05:36.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Object prefix: benchmark_data_smithi060_30537 2024-09-15T03:05:36.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:05:36.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 0 0 0 0 0 0 - 0 2024-09-15T03:05:36.705 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-15T03:05:36.696+0000 7f50cedec700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-15T03:05:37.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 1 20 1657 1637 12.7884 12.7891 0.0172393 0.0121527 2024-09-15T03:05:38.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 2 20 3269 3249 12.6905 12.5938 0.0108446 0.0122765 2024-09-15T03:05:39.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 3 20 4796 4776 12.4366 11.9297 0.0078859 0.0125103 2024-09-15T03:05:40.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 4 20 6289 6269 12.2432 11.6641 0.0143544 0.0127347 2024-09-15T03:05:41.348 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 5 20 7708 7688 12.0113 11.0859 0.0090973 0.0129865 2024-09-15T03:05:42.349 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 6 20 8829 8809 11.4679 8.75781 0.0324107 0.0136011 2024-09-15T03:05:43.349 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 7 19 10184 10165 11.3429 10.5938 0.0183241 0.013755 2024-09-15T03:05:44.349 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 8 20 11502 11482 11.2105 10.2891 0.0126476 0.0139176 2024-09-15T03:05:45.349 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 9 20 12779 12759 11.0733 9.97656 0.0242605 0.0140981 2024-09-15T03:05:46.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:05:46.349 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 10 20 13999 13979 10.919 9.53125 0.0137049 0.0142863 2024-09-15T03:05:46.827 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:46.836 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:05:46.854 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-15T03:05:46.641325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95214140,"num_objects":11627,"num_object_clones":0,"num_object_copies":46508,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":11681,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":11627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11654,"num_write_kb":93114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11654,"ondisk_log_size":11654,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1338036,"kb_used_data":126864,"kb_used_omap":12,"kb_used_meta":1211123,"kb_avail":376149324,"statfs":{"total":386547056640,"available":385176907776,"internally_reserved":0,"allocated":129908736,"data_stored":129380438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1240190770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[23,4,7,6,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":95019008,"num_objects":11599,"num_object_clones":0,"num_object_copies":46396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11599,"num_write_kb":92792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006791"},"pg_stats":[{"pgid":"1.f","version":"27'757","reported_seq":786,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:45.043030+0000","last_change":"2024-09-15T03:05:36.363087+0000","last_active":"2024-09-15T03:05:45.043030+0000","last_peered":"2024-09-15T03:05:45.043030+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.078107+0000","last_became_peered":"2024-09-15T03:05:27.078107+0000","last_unstale":"2024-09-15T03:05:45.043030+0000","last_undegraded":"2024-09-15T03:05:36.359817+0000","last_fullsized":"2024-09-15T03:05:27.059184+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:26.729372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6201344,"num_objects":757,"num_object_clones":0,"num_object_copies":3028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":757,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":757,"num_write_kb":6056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'813","reported_seq":853,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.918746+0000","last_change":"2024-09-15T03:05:27.084884+0000","last_active":"2024-09-15T03:05:45.918746+0000","last_peered":"2024-09-15T03:05:45.918746+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:05:27.076040+0000","last_became_peered":"2024-09-15T03:05:27.076040+0000","last_unstale":"2024-09-15T03:05:45.918746+0000","last_undegraded":"2024-09-15T03:05:27.063502+0000","last_fullsized":"2024-09-15T03:05:27.062931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":813,"log_dups_size":0,"ondisk_log_size":813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:49:47.581517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6641266,"num_objects":811,"num_object_clones":0,"num_object_copies":3244,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":815,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":813,"num_write_kb":6500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":809},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'503","reported_seq":561,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:41.687801+0000","last_change":"2024-09-15T03:05:27.076017+0000","last_active":"2024-09-15T03:05:41.687801+0000","last_peered":"2024-09-15T03:05:41.687801+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:05:27.073076+0000","last_became_peered":"2024-09-15T03:05:27.073076+0000","last_unstale":"2024-09-15T03:05:41.687801+0000","last_undegraded":"2024-09-15T03:05:27.062455+0000","last_fullsized":"2024-09-15T03:05:27.061642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":503,"log_dups_size":0,"ondisk_log_size":503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:12.173681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4102969,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":503,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":503,"num_write_kb":4014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":500},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'777","reported_seq":817,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.949296+0000","last_change":"2024-09-15T03:05:27.090005+0000","last_active":"2024-09-15T03:05:45.949296+0000","last_peered":"2024-09-15T03:05:45.949296+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:05:27.077226+0000","last_became_peered":"2024-09-15T03:05:27.077226+0000","last_unstale":"2024-09-15T03:05:45.949296+0000","last_undegraded":"2024-09-15T03:05:27.064312+0000","last_fullsized":"2024-09-15T03:05:27.063909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:18:26.641641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6346354,"num_objects":775,"num_object_clones":0,"num_object_copies":3100,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":777,"num_write_kb":6205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":773},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'808","reported_seq":838,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.060934+0000","last_change":"2024-09-15T03:05:27.085785+0000","last_active":"2024-09-15T03:05:45.060934+0000","last_peered":"2024-09-15T03:05:45.060934+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:05:27.070138+0000","last_became_peered":"2024-09-15T03:05:27.070138+0000","last_unstale":"2024-09-15T03:05:45.060934+0000","last_undegraded":"2024-09-15T03:05:27.057975+0000","last_fullsized":"2024-09-15T03:05:27.057143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:39:19.992759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6608498,"num_objects":807,"num_object_clones":0,"num_object_copies":3228,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":808,"num_write_kb":6461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":805},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'800","reported_seq":829,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:45.009153+0000","last_change":"2024-09-15T03:05:36.377095+0000","last_active":"2024-09-15T03:05:45.009153+0000","last_peered":"2024-09-15T03:05:45.009153+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.069510+0000","last_became_peered":"2024-09-15T03:05:27.069510+0000","last_unstale":"2024-09-15T03:05:45.009153+0000","last_undegraded":"2024-09-15T03:05:36.359227+0000","last_fullsized":"2024-09-15T03:05:27.057867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":800,"log_dups_size":0,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:56.697498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":800,"num_object_clones":0,"num_object_copies":3200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'862","reported_seq":902,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.934384+0000","last_change":"2024-09-15T03:05:27.081660+0000","last_active":"2024-09-15T03:05:45.934384+0000","last_peered":"2024-09-15T03:05:45.934384+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:05:27.073004+0000","last_became_peered":"2024-09-15T03:05:27.073004+0000","last_unstale":"2024-09-15T03:05:45.934384+0000","last_undegraded":"2024-09-15T03:05:27.062946+0000","last_fullsized":"2024-09-15T03:05:27.061959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":862,"log_dups_size":0,"ondisk_log_size":862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:02:15.775095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7026290,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":862,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":862,"num_write_kb":6876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":856},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'857","reported_seq":897,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.946437+0000","last_change":"2024-09-15T03:05:27.088312+0000","last_active":"2024-09-15T03:05:45.946437+0000","last_peered":"2024-09-15T03:05:45.946437+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:05:27.076435+0000","last_became_peered":"2024-09-15T03:05:27.076435+0000","last_unstale":"2024-09-15T03:05:45.946437+0000","last_undegraded":"2024-09-15T03:05:27.064600+0000","last_fullsized":"2024-09-15T03:05:27.063343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":857,"log_dups_size":0,"ondisk_log_size":857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:05:12.788710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6993522,"num_objects":854,"num_object_clones":0,"num_object_copies":3416,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":858,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":857,"num_write_kb":6844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":852},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'802","reported_seq":831,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.044048+0000","last_change":"2024-09-15T03:05:27.091808+0000","last_active":"2024-09-15T03:05:45.044048+0000","last_peered":"2024-09-15T03:05:45.044048+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:05:27.079178+0000","last_became_peered":"2024-09-15T03:05:27.079178+0000","last_unstale":"2024-09-15T03:05:45.044048+0000","last_undegraded":"2024-09-15T03:05:27.058589+0000","last_fullsized":"2024-09-15T03:05:27.057594+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":802,"log_dups_size":0,"ondisk_log_size":802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:03:17.277311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6524132,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":803,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":802,"num_write_kb":6393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":794},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'500","reported_seq":557,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:41.639767+0000","last_change":"2024-09-15T03:05:27.079875+0000","last_active":"2024-09-15T03:05:41.639767+0000","last_peered":"2024-09-15T03:05:41.639767+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:05:27.073080+0000","last_became_peered":"2024-09-15T03:05:27.073080+0000","last_unstale":"2024-09-15T03:05:41.639767+0000","last_undegraded":"2024-09-15T03:05:27.060756+0000","last_fullsized":"2024-09-15T03:05:27.059796+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:05:43.290430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4093554,"num_objects":500,"num_object_clones":0,"num_object_copies":2000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":504,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":500,"num_write_kb":3998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":498},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'782","reported_seq":811,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.005088+0000","last_change":"2024-09-15T03:05:27.082524+0000","last_active":"2024-09-15T03:05:45.005088+0000","last_peered":"2024-09-15T03:05:45.005088+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:05:27.069721+0000","last_became_peered":"2024-09-15T03:05:27.069721+0000","last_unstale":"2024-09-15T03:05:45.005088+0000","last_undegraded":"2024-09-15T03:05:27.058374+0000","last_fullsized":"2024-09-15T03:05:27.057414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":782,"log_dups_size":0,"ondisk_log_size":782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:04:25.934882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6386091,"num_objects":780,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":786,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":782,"num_write_kb":6244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":777},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'746","reported_seq":763,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.060559+0000","last_change":"2024-09-15T03:05:27.088981+0000","last_active":"2024-09-15T03:05:45.060559+0000","last_peered":"2024-09-15T03:05:45.060559+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:05:27.070691+0000","last_became_peered":"2024-09-15T03:05:27.070691+0000","last_unstale":"2024-09-15T03:05:45.060559+0000","last_undegraded":"2024-09-15T03:05:27.064197+0000","last_fullsized":"2024-09-15T03:05:27.063869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":746,"log_dups_size":0,"ondisk_log_size":746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:19:59.460163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6110009,"num_objects":746,"num_object_clones":0,"num_object_copies":2984,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":748,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":5967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":745},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'493","reported_seq":550,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:41.678690+0000","last_change":"2024-09-15T03:05:27.083985+0000","last_active":"2024-09-15T03:05:41.678690+0000","last_peered":"2024-09-15T03:05:41.678690+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:05:27.073296+0000","last_became_peered":"2024-09-15T03:05:27.073296+0000","last_unstale":"2024-09-15T03:05:41.678690+0000","last_undegraded":"2024-09-15T03:05:27.065069+0000","last_fullsized":"2024-09-15T03:05:27.064221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":493,"log_dups_size":0,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:22:11.122806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4037433,"num_objects":493,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":495,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":493,"num_write_kb":3943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":492},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'757","reported_seq":784,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.045229+0000","last_change":"2024-09-15T03:05:27.090687+0000","last_active":"2024-09-15T03:05:45.045229+0000","last_peered":"2024-09-15T03:05:45.045229+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:05:27.071179+0000","last_became_peered":"2024-09-15T03:05:27.071179+0000","last_unstale":"2024-09-15T03:05:45.045229+0000","last_undegraded":"2024-09-15T03:05:27.059665+0000","last_fullsized":"2024-09-15T03:05:27.057178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:36:34.628592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6182514,"num_objects":755,"num_object_clones":0,"num_object_copies":3020,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":759,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":757,"num_write_kb":6052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":753},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'477","reported_seq":533,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:41.678027+0000","last_change":"2024-09-15T03:05:27.087543+0000","last_active":"2024-09-15T03:05:41.678027+0000","last_peered":"2024-09-15T03:05:41.678027+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:05:27.074164+0000","last_became_peered":"2024-09-15T03:05:27.074164+0000","last_unstale":"2024-09-15T03:05:41.678027+0000","last_undegraded":"2024-09-15T03:05:27.063361+0000","last_fullsized":"2024-09-15T03:05:27.062449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:45:22.140009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3896946,"num_objects":476,"num_object_clones":0,"num_object_copies":1904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":480,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":477,"num_write_kb":3813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":474},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'920","reported_seq":960,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T03:05:45.955634+0000","last_change":"2024-09-15T03:05:27.086812+0000","last_active":"2024-09-15T03:05:45.955634+0000","last_peered":"2024-09-15T03:05:45.955634+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:05:27.076127+0000","last_became_peered":"2024-09-15T03:05:27.076127+0000","last_unstale":"2024-09-15T03:05:45.955634+0000","last_undegraded":"2024-09-15T03:05:27.063025+0000","last_fullsized":"2024-09-15T03:05:27.061985+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":920,"log_dups_size":0,"ondisk_log_size":920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:21:26.414306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7509618,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":921,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":7348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":915},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95214140,"num_objects":11627,"num_object_clones":0,"num_object_copies":46508,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":11681,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":11627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11654,"num_write_kb":93114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":138981376,"data_stored":138981376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11654,"ondisk_log_size":11654,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":351620,"kb_used_data":49596,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94020220,"statfs":{"total":96636764160,"available":96276705280,"internally_reserved":0,"allocated":50786304,"data_stored":50642715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,0,1,2,1,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.187}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22500000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337248,"kb_used_data":27416,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94034592,"statfs":{"total":96636764160,"available":96291422208,"internally_reserved":0,"allocated":28073984,"data_stored":27933541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]}]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":188416,"data_stored":87739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":351628,"kb_used_data":49668,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":94020212,"statfs":{"total":96636764160,"available":96276697088,"internally_reserved":0,"allocated":50860032,"data_stored":50716443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,4,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55492608,"data_stored":55492608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32518144,"data_stored":32518144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50929664,"data_stored":50929664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:05:46.855 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2024-09-15T03:05:46.855 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:46.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2024-09-15T03:05:47.113 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:47.113 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:47.114 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_4", 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 1224592654, 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'3", 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'1", 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:47.115 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_6", 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 3101443614, 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:47.116 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'4", 2024-09-15T03:05:47.117 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'2", 2024-09-15T03:05:47.117 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:47.117 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:47.117 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:47.117 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:47.117 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:47.117 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(0)", 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:47.118 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:47.130 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T03:05:47.131 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2024-09-15T03:05:47.131 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2024-09-15T03:05:47.350 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 11 20 15077 15057 10.6918 8.42188 0.0213657 0.0145949 2024-09-15T03:05:47.396 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:47.431 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:47.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2024-09-15T03:05:47.676 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 1, 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 1, 2024-09-15T03:05:47.677 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_7", 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 2358579165, 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'3", 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:47.678 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: "2(0)" 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(1)", 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:47.679 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:47.699 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-15T03:05:47.700 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2024-09-15T03:05:47.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2024-09-15T03:05:47.982 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-15T03:05:47.994 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:47.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2024-09-15T03:05:48.274 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:48.274 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_5", 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 4248945696, 2024-09-15T03:05:48.275 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'3", 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:48.276 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_5", 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 4212789968, 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'4", 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:48.277 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(0)", 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:48.278 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:48.279 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:48.279 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:48.313 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T03:05:48.313 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2024-09-15T03:05:48.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2024-09-15T03:05:48.350 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 12 20 16155 16135 10.5026 8.42188 0.014904 0.0148581 2024-09-15T03:05:48.637 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:48.665 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:48.665 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2024-09-15T03:05:48.879 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:48.879 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:48.879 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:48.879 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_3", 2024-09-15T03:05:48.880 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 459768081, 2024-09-15T03:05:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:48.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:48.882 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'3", 2024-09-15T03:05:48.882 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'1", 2024-09-15T03:05:48.882 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:48.882 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:48.883 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_3", 2024-09-15T03:05:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 480221105, 2024-09-15T03:05:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'2", 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:48.885 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:48.886 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(1)", 2024-09-15T03:05:48.887 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:48.887 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:48.887 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:48.887 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:48.887 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:48.901 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-15T03:05:48.901 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2024-09-15T03:05:48.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2024-09-15T03:05:49.206 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:49.214 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2024-09-15T03:05:49.214 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:49.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2024-09-15T03:05:49.351 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 13 20 17196 17176 10.3192 8.13281 0.0175096 0.0151235 2024-09-15T03:05:49.438 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:49.438 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:49.439 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_6", 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 1708673474, 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'5", 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:49.440 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_9", 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 831402034, 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:49.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'6", 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:49.442 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(2)", 2024-09-15T03:05:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:49.443 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:49.443 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:49.455 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-15T03:05:49.455 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2024-09-15T03:05:49.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2024-09-15T03:05:49.686 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:49.705 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:49.705 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2024-09-15T03:05:49.949 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:49.949 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:49.949 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:49.949 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:49.949 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:49.949 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_8", 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 1901787523, 2024-09-15T03:05:49.950 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'5", 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'3", 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:49.951 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_1", 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 3122379363, 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'4", 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:49.952 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(0)", 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:49.953 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:49.967 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "16'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T03:05:49.967 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2024-09-15T03:05:49.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2024-09-15T03:05:50.231 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:50.253 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:50.254 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2024-09-15T03:05:50.351 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 14 20 18190 18170 10.1368 7.76562 0.0246255 0.0153948 2024-09-15T03:05:50.545 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:50.545 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:50.545 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:50.545 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:50.545 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 3, 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 3, 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_4", 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:50.546 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 873891716, 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'8", 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:50.547 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_7", 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 3996155460, 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'9", 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'6", 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:50.548 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_2", 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 1767667508, 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'7", 2024-09-15T03:05:50.549 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(3)", 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:50.550 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:50.551 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:50.551 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:50.551 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:50.566 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'9", 'have': "16'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-15T03:05:50.566 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2024-09-15T03:05:50.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2024-09-15T03:05:50.727 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-15T03:05:50.721+0000 7f50cedec700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 13 pgs recovery_unfound (PG_DAMAGED) 2024-09-15T03:05:50.874 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-15T03:05:50.891 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:50.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.118 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_1", 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 26281365, 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.119 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'1", 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "2(0)" 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_2", 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 3196340981, 2024-09-15T03:05:51.120 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'2", 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "2(0)" 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:51.121 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:51.122 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:51.122 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:51.122 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(3)", 2024-09-15T03:05:51.122 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:51.122 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:51.122 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:51.122 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:51.122 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:51.138 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-15T03:05:51.138 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2024-09-15T03:05:51.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2024-09-15T03:05:51.351 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 15 20 19211 19191 9.99273 7.97656 0.01873 0.0156257 2024-09-15T03:05:51.458 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:51.471 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:51.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2024-09-15T03:05:51.705 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:51.705 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 3, 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 3, 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:51.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_8", 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 1861424966, 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:51.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'5", 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_6", 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:51.708 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 3910131350, 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'4", 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:51.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_4", 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 2787839990, 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:51.710 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'3", 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: "2(3)" 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:51.711 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:51.712 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(2)", 2024-09-15T03:05:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:51.713 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:51.713 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:51.726 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-15T03:05:51.726 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2024-09-15T03:05:51.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2024-09-15T03:05:52.113 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-15T03:05:52.132 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:52.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2024-09-15T03:05:52.351 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 16 20 20106 20086 9.80519 6.99219 0.0191713 0.0159194 2024-09-15T03:05:52.353 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:52.353 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:52.353 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:52.353 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:52.353 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:52.353 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 1, 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 1, 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_7", 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 3761017527, 2024-09-15T03:05:52.354 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'1", 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:52.355 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(0)", 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:52.356 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:52.380 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T03:05:52.380 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2024-09-15T03:05:52.380 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2024-09-15T03:05:52.665 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-15T03:05:52.678 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:52.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2024-09-15T03:05:52.893 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:52.893 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:52.893 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:52.894 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:52.894 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:52.894 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:52.894 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:52.894 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:52.895 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:52.895 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:52.895 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 1, 2024-09-15T03:05:52.895 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 1, 2024-09-15T03:05:52.895 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:52.895 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:52.896 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:52.896 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_8", 2024-09-15T03:05:52.896 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:52.896 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 2730327960, 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'1", 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: "2(1)" 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:52.897 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:52.898 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:52.898 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:52.898 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:52.898 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:52.898 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:52.899 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(0)", 2024-09-15T03:05:52.899 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:52.899 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:52.899 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:52.899 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:52.899 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:52.916 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T03:05:52.916 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2024-09-15T03:05:52.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2024-09-15T03:05:53.206 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-15T03:05:53.226 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:53.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2024-09-15T03:05:53.351 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 17 20 21076 21056 9.67417 7.57812 0.0292705 0.016139 2024-09-15T03:05:53.476 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:53.476 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:53.477 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_5", 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 2998549577, 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'4", 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'2", 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:53.478 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_2", 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 4032093161, 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:53.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'3", 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'1", 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:53.480 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:53.481 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(0)", 2024-09-15T03:05:53.481 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:53.481 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:53.481 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:53.481 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:53.481 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:53.507 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T03:05:53.507 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2024-09-15T03:05:53.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2024-09-15T03:05:53.789 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:53.809 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:53.809 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2024-09-15T03:05:54.015 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:54.015 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:54.015 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:54.015 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:54.015 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:54.015 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_9", 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:54.016 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 1217469130, 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'3", 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'1", 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "2(1)" 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:54.017 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "new_9", 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 170713514, 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'2", 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:54.018 INFO:teuthology.orchestra.run.smithi060.stdout: "2(1)" 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(0)", 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:54.019 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:54.030 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T03:05:54.030 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2024-09-15T03:05:54.030 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2024-09-15T03:05:54.306 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:54.329 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2024-09-15T03:05:54.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2024-09-15T03:05:54.351 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 18 20 22038 22018 9.55424 7.51562 0.014847 0.0163427 2024-09-15T03:05:54.558 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "offset": { 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "", 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": 0, 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 0, 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": -9223372036854775808, 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "num_missing": 2, 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "num_unfound": 2, 2024-09-15T03:05:54.559 INFO:teuthology.orchestra.run.smithi060.stdout: "objects": [ 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existed_3", 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 943960171, 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'5", 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "0'0", 2024-09-15T03:05:54.560 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": { 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "oid": "existing_1", 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "key": "", 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "snapid": -2, 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "hash": 2885884443, 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 0, 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "pool": 1, 2024-09-15T03:05:54.561 INFO:teuthology.orchestra.run.smithi060.stdout: "namespace": "" 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "need": "19'4", 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "have": "16'1", 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": "none", 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "clean_regions": "clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0", 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "locations": [ 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "2(2)" 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "NotRecovering", 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "available_might_have_unfound": true, 2024-09-15T03:05:54.562 INFO:teuthology.orchestra.run.smithi060.stdout: "might_have_unfound": [ 2024-09-15T03:05:54.563 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-09-15T03:05:54.563 INFO:teuthology.orchestra.run.smithi060.stdout: "osd": "1(0)", 2024-09-15T03:05:54.563 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "osd is down" 2024-09-15T03:05:54.563 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-09-15T03:05:54.563 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-09-15T03:05:54.563 INFO:teuthology.orchestra.run.smithi060.stdout: "more": false 2024-09-15T03:05:54.563 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:05:54.573 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: 0, new_object: 1', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [6969~18446744073709544646], clean_omap: 1, new_object: 0', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T03:05:54.573 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2024-09-15T03:05:54.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2024-09-15T03:05:54.702 INFO:teuthology.orchestra.run.smithi060.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T03:05:54.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-15T03:05:54.989 INFO:teuthology.orchestra.run.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-15T03:05:55.013 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-15T03:05:55.349 INFO:teuthology.orchestra.run.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-15T03:05:55.352 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 19 20 22833 22813 9.37795 6.21094 0.0344806 0.0166509 2024-09-15T03:05:55.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2024-09-15T03:05:55.642 INFO:teuthology.orchestra.run.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-15T03:05:55.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:05:55.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:05:55.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:05:56.023 INFO:teuthology.orchestra.run.smithi060.stdout:111669149706 2024-09-15T03:05:56.024 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:05:56.039 INFO:teuthology.orchestra.run.smithi060.stdout:38654705682 2024-09-15T03:05:56.040 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:05:56.044 INFO:teuthology.orchestra.run.smithi060.stdout:98784247820 2024-09-15T03:05:56.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:05:56.352 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:05:56.350767+0000 min lat: 0.00240791 max lat: 0.226371 avg lat: 0.0168821 2024-09-15T03:05:56.352 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:05:56.352 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 20 20 23688 23668 9.24303 6.67969 0.0228613 0.0168821 2024-09-15T03:05:56.686 INFO:teuthology.orchestra.run.smithi060.stdout:98784247818 2024-09-15T03:05:56.705 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-15T03:05:56.697+0000 7f50cedec700 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 7 pgs recovery_unfound (PG_DAMAGED) 2024-09-15T03:05:56.722 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247820 got 98784247818 for osd.0 2024-09-15T03:05:56.736 INFO:teuthology.orchestra.run.smithi060.stdout:111669149704 2024-09-15T03:05:56.753 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149704 for osd.3 2024-09-15T03:05:56.797 INFO:teuthology.orchestra.run.smithi060.stdout:38654705682 2024-09-15T03:05:56.821 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705682 got 38654705682 for osd.2 2024-09-15T03:05:56.821 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:57.352 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 21 20 24462 24442 9.09083 6.04688 0.0177506 0.017178 2024-09-15T03:05:57.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:05:57.754 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:05:58.352 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 22 20 25337 25317 8.98832 6.83594 0.0201041 0.0173669 2024-09-15T03:05:58.355 INFO:teuthology.orchestra.run.smithi060.stdout:98784247820 2024-09-15T03:05:58.381 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247820 got 98784247820 for osd.0 2024-09-15T03:05:58.381 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:58.536 INFO:teuthology.orchestra.run.smithi060.stdout:111669149706 2024-09-15T03:05:58.557 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149706 for osd.3 2024-09-15T03:05:58.558 DEBUG:teuthology.parallel:result is None 2024-09-15T03:05:58.558 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-15T03:05:58.558 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:05:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:59.203 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:05:59.220 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-09-15T03:05:58.645853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":192961084,"num_objects":23559,"num_object_clones":0,"num_object_copies":94236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23586,"num_write_kb":188570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23613,"ondisk_log_size":23613,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1483956,"kb_used_data":276944,"kb_used_omap":12,"kb_used_meta":1206963,"kb_avail":376003404,"statfs":{"total":386547056640,"available":385027485696,"internally_reserved":0,"allocated":283590656,"data_stored":283062358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1235930930},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,11,12,26,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":97746944,"num_objects":11932,"num_object_clones":0,"num_object_copies":47728,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":11878,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":11932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":95456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004528"},"pg_stats":[{"pgid":"1.f","version":"27'1404","reported_seq":1433,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.928928+0000","last_change":"2024-09-15T03:05:36.363087+0000","last_active":"2024-09-15T03:05:55.928928+0000","last_peered":"2024-09-15T03:05:55.928928+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.078107+0000","last_became_peered":"2024-09-15T03:05:27.078107+0000","last_unstale":"2024-09-15T03:05:55.928928+0000","last_undegraded":"2024-09-15T03:05:36.359817+0000","last_fullsized":"2024-09-15T03:05:27.059184+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1404,"log_dups_size":0,"ondisk_log_size":1404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:26.729372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11501568,"num_objects":1404,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1404,"num_write_kb":11232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1438","reported_seq":1488,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.951369+0000","last_change":"2024-09-15T03:05:47.437364+0000","last_active":"2024-09-15T03:05:55.951369+0000","last_peered":"2024-09-15T03:05:55.951369+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:05:27.076040+0000","last_became_peered":"2024-09-15T03:05:27.076040+0000","last_unstale":"2024-09-15T03:05:55.951369+0000","last_undegraded":"2024-09-15T03:05:27.063502+0000","last_fullsized":"2024-09-15T03:05:27.062931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11744882,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1436,"num_write_kb":11484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1543","reported_seq":1608,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:56.675783+0000","last_change":"2024-09-15T03:05:47.986686+0000","last_active":"2024-09-15T03:05:56.675783+0000","last_peered":"2024-09-15T03:05:56.675783+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:05:27.073076+0000","last_became_peered":"2024-09-15T03:05:27.073076+0000","last_unstale":"2024-09-15T03:05:56.675783+0000","last_undegraded":"2024-09-15T03:05:27.062455+0000","last_fullsized":"2024-09-15T03:05:27.061642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12614457,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1542,"num_write_kb":12326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1369","reported_seq":1419,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.951068+0000","last_change":"2024-09-15T03:05:48.701594+0000","last_active":"2024-09-15T03:05:55.951068+0000","last_peered":"2024-09-15T03:05:55.951068+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:05:27.077226+0000","last_became_peered":"2024-09-15T03:05:27.077226+0000","last_unstale":"2024-09-15T03:05:55.951068+0000","last_undegraded":"2024-09-15T03:05:27.064312+0000","last_fullsized":"2024-09-15T03:05:27.063909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1369,"log_dups_size":0,"ondisk_log_size":1369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11179634,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":10925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1365}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1493","reported_seq":1533,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.964423+0000","last_change":"2024-09-15T03:05:49.245426+0000","last_active":"2024-09-15T03:05:55.964423+0000","last_peered":"2024-09-15T03:05:55.964423+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:05:27.070138+0000","last_became_peered":"2024-09-15T03:05:27.070138+0000","last_unstale":"2024-09-15T03:05:55.964423+0000","last_undegraded":"2024-09-15T03:05:27.057975+0000","last_fullsized":"2024-09-15T03:05:27.057143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1493,"log_dups_size":0,"ondisk_log_size":1493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203634,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1491,"num_write_kb":11925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1488","reported_seq":1517,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.973483+0000","last_change":"2024-09-15T03:05:36.377095+0000","last_active":"2024-09-15T03:05:55.973483+0000","last_peered":"2024-09-15T03:05:55.973483+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.069510+0000","last_became_peered":"2024-09-15T03:05:27.069510+0000","last_unstale":"2024-09-15T03:05:55.973483+0000","last_undegraded":"2024-09-15T03:05:36.359227+0000","last_fullsized":"2024-09-15T03:05:27.057867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1488,"log_dups_size":0,"ondisk_log_size":1488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:56.697498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":1488,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":11904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1481","reported_seq":1528,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.984438+0000","last_change":"2024-09-15T03:05:49.758375+0000","last_active":"2024-09-15T03:05:55.984438+0000","last_peered":"2024-09-15T03:05:55.984438+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:05:27.073004+0000","last_became_peered":"2024-09-15T03:05:27.073004+0000","last_unstale":"2024-09-15T03:05:55.984438+0000","last_undegraded":"2024-09-15T03:05:27.062946+0000","last_fullsized":"2024-09-15T03:05:27.061959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12080754,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1469","reported_seq":1519,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.956669+0000","last_change":"2024-09-15T03:05:50.289840+0000","last_active":"2024-09-15T03:05:55.956669+0000","last_peered":"2024-09-15T03:05:55.956669+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:05:27.076435+0000","last_became_peered":"2024-09-15T03:05:27.076435+0000","last_unstale":"2024-09-15T03:05:55.956669+0000","last_undegraded":"2024-09-15T03:05:27.064600+0000","last_fullsized":"2024-09-15T03:05:27.063343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11990642,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1467,"num_write_kb":11724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1464}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1490","reported_seq":1532,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.956128+0000","last_change":"2024-09-15T03:05:50.981389+0000","last_active":"2024-09-15T03:05:55.956128+0000","last_peered":"2024-09-15T03:05:55.956128+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:05:27.079178+0000","last_became_peered":"2024-09-15T03:05:27.079178+0000","last_unstale":"2024-09-15T03:05:55.956128+0000","last_undegraded":"2024-09-15T03:05:27.058589+0000","last_fullsized":"2024-09-15T03:05:27.057594+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12135652,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1487,"num_write_kb":11873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1526","reported_seq":1593,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:56.694245+0000","last_change":"2024-09-15T03:05:51.509375+0000","last_active":"2024-09-15T03:05:56.694245+0000","last_peered":"2024-09-15T03:05:56.694245+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:05:27.073080+0000","last_became_peered":"2024-09-15T03:05:27.073080+0000","last_unstale":"2024-09-15T03:05:56.694245+0000","last_undegraded":"2024-09-15T03:05:27.060756+0000","last_fullsized":"2024-09-15T03:05:27.059796+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12482162,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1524}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1502","reported_seq":1544,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.940002+0000","last_change":"2024-09-15T03:05:52.241020+0000","last_active":"2024-09-15T03:05:55.940002+0000","last_peered":"2024-09-15T03:05:55.940002+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:05:27.069721+0000","last_became_peered":"2024-09-15T03:05:27.069721+0000","last_unstale":"2024-09-15T03:05:55.940002+0000","last_undegraded":"2024-09-15T03:05:27.058374+0000","last_fullsized":"2024-09-15T03:05:27.057414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12259755,"num_objects":1497,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1499,"num_write_kb":11980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1473","reported_seq":1498,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.984151+0000","last_change":"2024-09-15T03:05:52.701122+0000","last_active":"2024-09-15T03:05:55.984151+0000","last_peered":"2024-09-15T03:05:55.984151+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:05:27.070691+0000","last_became_peered":"2024-09-15T03:05:27.070691+0000","last_unstale":"2024-09-15T03:05:55.984151+0000","last_undegraded":"2024-09-15T03:05:27.064197+0000","last_fullsized":"2024-09-15T03:05:27.063869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12057401,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1472,"num_write_kb":11775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1515","reported_seq":1579,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:56.696659+0000","last_change":"2024-09-15T03:05:53.237197+0000","last_active":"2024-09-15T03:05:56.696659+0000","last_peered":"2024-09-15T03:05:56.696659+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:05:27.073296+0000","last_became_peered":"2024-09-15T03:05:27.073296+0000","last_unstale":"2024-09-15T03:05:56.696659+0000","last_undegraded":"2024-09-15T03:05:27.065069+0000","last_fullsized":"2024-09-15T03:05:27.064221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12401465,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1514,"num_write_kb":12111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1514}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1462","reported_seq":1499,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.986157+0000","last_change":"2024-09-15T03:05:53.844392+0000","last_active":"2024-09-15T03:05:55.986157+0000","last_peered":"2024-09-15T03:05:55.986157+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:05:27.071179+0000","last_became_peered":"2024-09-15T03:05:27.071179+0000","last_unstale":"2024-09-15T03:05:55.986157+0000","last_undegraded":"2024-09-15T03:05:27.059665+0000","last_fullsized":"2024-09-15T03:05:27.057178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941490,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":11676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1475","reported_seq":1543,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:56.698761+0000","last_change":"2024-09-15T03:05:54.363436+0000","last_active":"2024-09-15T03:05:56.698761+0000","last_peered":"2024-09-15T03:05:56.698761+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:05:27.074164+0000","last_became_peered":"2024-09-15T03:05:27.074164+0000","last_unstale":"2024-09-15T03:05:56.698761+0000","last_undegraded":"2024-09-15T03:05:27.063361+0000","last_fullsized":"2024-09-15T03:05:27.062449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12056178,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":11781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1485","reported_seq":1535,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.956872+0000","last_change":"2024-09-15T03:05:54.735419+0000","last_active":"2024-09-15T03:05:55.956872+0000","last_peered":"2024-09-15T03:05:55.956872+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:05:27.076127+0000","last_became_peered":"2024-09-15T03:05:27.076127+0000","last_unstale":"2024-09-15T03:05:55.956872+0000","last_undegraded":"2024-09-15T03:05:27.063025+0000","last_fullsized":"2024-09-15T03:05:27.061985+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12121714,"num_objects":1480,"num_object_clones":0,"num_object_copies":5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1483,"num_write_kb":11852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1480}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":192961084,"num_objects":23559,"num_object_clones":0,"num_object_copies":94236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23586,"num_write_kb":188570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289812480,"data_stored":289812480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23613,"ondisk_log_size":23613,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392656,"kb_used_data":90632,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93979184,"statfs":{"total":96636764160,"available":96234684416,"internally_reserved":0,"allocated":92807168,"data_stored":92663579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,3,5,14,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401552,"kb_used_data":95880,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93970288,"statfs":{"total":96636764160,"available":96225574912,"internally_reserved":0,"allocated":98181120,"data_stored":98040677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,1,12,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]}]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":188416,"data_stored":87739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392208,"kb_used_data":90248,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93979632,"statfs":{"total":96636764160,"available":96235143168,"internally_reserved":0,"allocated":92413952,"data_stored":92270363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95846400,"data_stored":95846400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98111488,"data_stored":98111488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95813632,"data_stored":95813632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:05:59.221 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:05:59.352 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 23 20 26212 26192 8.89473 6.83594 0.0231812 0.0175547 2024-09-15T03:05:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:05:59.736 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:05:59.750 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-09-15T03:05:58.645853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":192961084,"num_objects":23559,"num_object_clones":0,"num_object_copies":94236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23586,"num_write_kb":188570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23613,"ondisk_log_size":23613,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1483956,"kb_used_data":276944,"kb_used_omap":12,"kb_used_meta":1206963,"kb_avail":376003404,"statfs":{"total":386547056640,"available":385027485696,"internally_reserved":0,"allocated":283590656,"data_stored":283062358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12494,"internal_metadata":1235930930},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,11,12,26,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":97746944,"num_objects":11932,"num_object_clones":0,"num_object_copies":47728,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":11878,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":11932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":95456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004528"},"pg_stats":[{"pgid":"1.f","version":"27'1404","reported_seq":1433,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.928928+0000","last_change":"2024-09-15T03:05:36.363087+0000","last_active":"2024-09-15T03:05:55.928928+0000","last_peered":"2024-09-15T03:05:55.928928+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.078107+0000","last_became_peered":"2024-09-15T03:05:27.078107+0000","last_unstale":"2024-09-15T03:05:55.928928+0000","last_undegraded":"2024-09-15T03:05:36.359817+0000","last_fullsized":"2024-09-15T03:05:27.059184+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1404,"log_dups_size":0,"ondisk_log_size":1404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:26.729372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11501568,"num_objects":1404,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1404,"num_write_kb":11232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1438","reported_seq":1488,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.951369+0000","last_change":"2024-09-15T03:05:47.437364+0000","last_active":"2024-09-15T03:05:55.951369+0000","last_peered":"2024-09-15T03:05:55.951369+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:05:27.076040+0000","last_became_peered":"2024-09-15T03:05:27.076040+0000","last_unstale":"2024-09-15T03:05:55.951369+0000","last_undegraded":"2024-09-15T03:05:27.063502+0000","last_fullsized":"2024-09-15T03:05:27.062931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11744882,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1436,"num_write_kb":11484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1543","reported_seq":1608,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:56.675783+0000","last_change":"2024-09-15T03:05:47.986686+0000","last_active":"2024-09-15T03:05:56.675783+0000","last_peered":"2024-09-15T03:05:56.675783+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:05:27.073076+0000","last_became_peered":"2024-09-15T03:05:27.073076+0000","last_unstale":"2024-09-15T03:05:56.675783+0000","last_undegraded":"2024-09-15T03:05:27.062455+0000","last_fullsized":"2024-09-15T03:05:27.061642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1543,"log_dups_size":0,"ondisk_log_size":1543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12614457,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1540,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1542,"num_write_kb":12326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1369","reported_seq":1419,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.951068+0000","last_change":"2024-09-15T03:05:48.701594+0000","last_active":"2024-09-15T03:05:55.951068+0000","last_peered":"2024-09-15T03:05:55.951068+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:05:27.077226+0000","last_became_peered":"2024-09-15T03:05:27.077226+0000","last_unstale":"2024-09-15T03:05:55.951068+0000","last_undegraded":"2024-09-15T03:05:27.064312+0000","last_fullsized":"2024-09-15T03:05:27.063909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1369,"log_dups_size":0,"ondisk_log_size":1369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11179634,"num_objects":1365,"num_object_clones":0,"num_object_copies":5460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":10925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1365}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1493","reported_seq":1533,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.964423+0000","last_change":"2024-09-15T03:05:49.245426+0000","last_active":"2024-09-15T03:05:55.964423+0000","last_peered":"2024-09-15T03:05:55.964423+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:05:27.070138+0000","last_became_peered":"2024-09-15T03:05:27.070138+0000","last_unstale":"2024-09-15T03:05:55.964423+0000","last_undegraded":"2024-09-15T03:05:27.057975+0000","last_fullsized":"2024-09-15T03:05:27.057143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1493,"log_dups_size":0,"ondisk_log_size":1493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203634,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1491,"num_write_kb":11925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1488","reported_seq":1517,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.973483+0000","last_change":"2024-09-15T03:05:36.377095+0000","last_active":"2024-09-15T03:05:55.973483+0000","last_peered":"2024-09-15T03:05:55.973483+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.069510+0000","last_became_peered":"2024-09-15T03:05:27.069510+0000","last_unstale":"2024-09-15T03:05:55.973483+0000","last_undegraded":"2024-09-15T03:05:36.359227+0000","last_fullsized":"2024-09-15T03:05:27.057867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1488,"log_dups_size":0,"ondisk_log_size":1488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:56.697498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":1488,"num_object_clones":0,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1488,"num_write_kb":11904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1481","reported_seq":1528,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.984438+0000","last_change":"2024-09-15T03:05:49.758375+0000","last_active":"2024-09-15T03:05:55.984438+0000","last_peered":"2024-09-15T03:05:55.984438+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:05:27.073004+0000","last_became_peered":"2024-09-15T03:05:27.073004+0000","last_unstale":"2024-09-15T03:05:55.984438+0000","last_undegraded":"2024-09-15T03:05:27.062946+0000","last_fullsized":"2024-09-15T03:05:27.061959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12080754,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1469","reported_seq":1519,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.956669+0000","last_change":"2024-09-15T03:05:50.289840+0000","last_active":"2024-09-15T03:05:55.956669+0000","last_peered":"2024-09-15T03:05:55.956669+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:05:27.076435+0000","last_became_peered":"2024-09-15T03:05:27.076435+0000","last_unstale":"2024-09-15T03:05:55.956669+0000","last_undegraded":"2024-09-15T03:05:27.064600+0000","last_fullsized":"2024-09-15T03:05:27.063343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11990642,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1467,"num_write_kb":11724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1464}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1490","reported_seq":1532,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.956128+0000","last_change":"2024-09-15T03:05:50.981389+0000","last_active":"2024-09-15T03:05:55.956128+0000","last_peered":"2024-09-15T03:05:55.956128+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:05:27.079178+0000","last_became_peered":"2024-09-15T03:05:27.079178+0000","last_unstale":"2024-09-15T03:05:55.956128+0000","last_undegraded":"2024-09-15T03:05:27.058589+0000","last_fullsized":"2024-09-15T03:05:27.057594+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1490,"log_dups_size":0,"ondisk_log_size":1490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12135652,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1487,"num_write_kb":11873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1526","reported_seq":1593,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:56.694245+0000","last_change":"2024-09-15T03:05:51.509375+0000","last_active":"2024-09-15T03:05:56.694245+0000","last_peered":"2024-09-15T03:05:56.694245+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:05:27.073080+0000","last_became_peered":"2024-09-15T03:05:27.073080+0000","last_unstale":"2024-09-15T03:05:56.694245+0000","last_undegraded":"2024-09-15T03:05:27.060756+0000","last_fullsized":"2024-09-15T03:05:27.059796+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12482162,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1524,"num_write_kb":12190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1524}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1502","reported_seq":1544,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.940002+0000","last_change":"2024-09-15T03:05:52.241020+0000","last_active":"2024-09-15T03:05:55.940002+0000","last_peered":"2024-09-15T03:05:55.940002+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:05:27.069721+0000","last_became_peered":"2024-09-15T03:05:27.069721+0000","last_unstale":"2024-09-15T03:05:55.940002+0000","last_undegraded":"2024-09-15T03:05:27.058374+0000","last_fullsized":"2024-09-15T03:05:27.057414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12259755,"num_objects":1497,"num_object_clones":0,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1497,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1499,"num_write_kb":11980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1473","reported_seq":1498,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.984151+0000","last_change":"2024-09-15T03:05:52.701122+0000","last_active":"2024-09-15T03:05:55.984151+0000","last_peered":"2024-09-15T03:05:55.984151+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:05:27.070691+0000","last_became_peered":"2024-09-15T03:05:27.070691+0000","last_unstale":"2024-09-15T03:05:55.984151+0000","last_undegraded":"2024-09-15T03:05:27.064197+0000","last_fullsized":"2024-09-15T03:05:27.063869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12057401,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1472,"num_write_kb":11775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1515","reported_seq":1579,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:56.696659+0000","last_change":"2024-09-15T03:05:53.237197+0000","last_active":"2024-09-15T03:05:56.696659+0000","last_peered":"2024-09-15T03:05:56.696659+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:05:27.073296+0000","last_became_peered":"2024-09-15T03:05:27.073296+0000","last_unstale":"2024-09-15T03:05:56.696659+0000","last_undegraded":"2024-09-15T03:05:27.065069+0000","last_fullsized":"2024-09-15T03:05:27.064221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12401465,"num_objects":1514,"num_object_clones":0,"num_object_copies":6056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1514,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1514,"num_write_kb":12111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1514}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1462","reported_seq":1499,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.986157+0000","last_change":"2024-09-15T03:05:53.844392+0000","last_active":"2024-09-15T03:05:55.986157+0000","last_peered":"2024-09-15T03:05:55.986157+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:05:27.071179+0000","last_became_peered":"2024-09-15T03:05:27.071179+0000","last_unstale":"2024-09-15T03:05:55.986157+0000","last_undegraded":"2024-09-15T03:05:27.059665+0000","last_fullsized":"2024-09-15T03:05:27.057178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941490,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":11676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1475","reported_seq":1543,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:56.698761+0000","last_change":"2024-09-15T03:05:54.363436+0000","last_active":"2024-09-15T03:05:56.698761+0000","last_peered":"2024-09-15T03:05:56.698761+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:05:27.074164+0000","last_became_peered":"2024-09-15T03:05:27.074164+0000","last_unstale":"2024-09-15T03:05:56.698761+0000","last_undegraded":"2024-09-15T03:05:27.063361+0000","last_fullsized":"2024-09-15T03:05:27.062449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12056178,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":11781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1485","reported_seq":1535,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:05:55.956872+0000","last_change":"2024-09-15T03:05:54.735419+0000","last_active":"2024-09-15T03:05:55.956872+0000","last_peered":"2024-09-15T03:05:55.956872+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:05:27.076127+0000","last_became_peered":"2024-09-15T03:05:27.076127+0000","last_unstale":"2024-09-15T03:05:55.956872+0000","last_undegraded":"2024-09-15T03:05:27.063025+0000","last_fullsized":"2024-09-15T03:05:27.061985+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12121714,"num_objects":1480,"num_object_clones":0,"num_object_copies":5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1483,"num_write_kb":11852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1480}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":192961084,"num_objects":23559,"num_object_clones":0,"num_object_copies":94236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23586,"num_write_kb":188570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":289812480,"data_stored":289812480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23613,"ondisk_log_size":23613,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392656,"kb_used_data":90632,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93979184,"statfs":{"total":96636764160,"available":96234684416,"internally_reserved":0,"allocated":92807168,"data_stored":92663579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,3,5,14,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401552,"kb_used_data":95880,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93970288,"statfs":{"total":96636764160,"available":96225574912,"internally_reserved":0,"allocated":98181120,"data_stored":98040677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,6,1,12,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]}]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297540,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074300,"statfs":{"total":96636764160,"available":96332083200,"internally_reserved":0,"allocated":188416,"data_stored":87739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392208,"kb_used_data":90248,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93979632,"statfs":{"total":96636764160,"available":96235143168,"internally_reserved":0,"allocated":92413952,"data_stored":92270363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95846400,"data_stored":95846400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98111488,"data_stored":98111488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":95813632,"data_stored":95813632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:05:59.752 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-15T03:05:59.752 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2024-09-15T03:05:59.752 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2024-09-15T03:05:59.896 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2024-09-15T03:05:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:05:59.907 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2024-09-15T03:05:59.907 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2024-09-15T03:06:00.070 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2024-09-15T03:06:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.081 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2024-09-15T03:06:00.081 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2024-09-15T03:06:00.187 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2024-09-15T03:06:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.201 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2024-09-15T03:06:00.201 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2024-09-15T03:06:00.286 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2024-09-15T03:06:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.297 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2024-09-15T03:06:00.297 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2024-09-15T03:06:00.352 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 24 20 27089 27069 8.80951 6.85156 0.00921317 0.0177242 2024-09-15T03:06:00.391 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2024-09-15T03:06:00.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.406 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2024-09-15T03:06:00.406 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2024-09-15T03:06:00.534 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2024-09-15T03:06:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.555 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2024-09-15T03:06:00.555 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2024-09-15T03:06:00.654 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2024-09-15T03:06:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.658 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2024-09-15T03:06:00.658 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2024-09-15T03:06:00.741 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2024-09-15T03:06:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.760 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2024-09-15T03:06:00.760 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2024-09-15T03:06:00.861 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2024-09-15T03:06:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.873 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2024-09-15T03:06:00.873 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2024-09-15T03:06:00.979 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2024-09-15T03:06:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:00.986 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2024-09-15T03:06:00.986 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2024-09-15T03:06:01.060 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2024-09-15T03:06:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:01.067 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2024-09-15T03:06:01.067 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2024-09-15T03:06:01.146 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2024-09-15T03:06:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:01.163 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2024-09-15T03:06:01.163 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2024-09-15T03:06:01.287 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2024-09-15T03:06:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:01.294 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2024-09-15T03:06:01.294 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2024-09-15T03:06:01.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 25 20 27957 27937 8.72816 6.78125 0.0211116 0.0178886 2024-09-15T03:06:01.391 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2024-09-15T03:06:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:01.407 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2024-09-15T03:06:01.407 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2024-09-15T03:06:01.479 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2024-09-15T03:06:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:01.490 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2024-09-15T03:06:01.490 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2024-09-15T03:06:01.579 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2024-09-15T03:06:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:01.583 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2024-09-15T03:06:01.583 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2024-09-15T03:06:01.689 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2024-09-15T03:06:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:01.747 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2024-09-15T03:06:01.747 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2024-09-15T03:06:01.858 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2024-09-15T03:06:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:01.873 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2024-09-15T03:06:01.873 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2024-09-15T03:06:02.034 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2024-09-15T03:06:02.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.040 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2024-09-15T03:06:02.040 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2024-09-15T03:06:02.133 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2024-09-15T03:06:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.136 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2024-09-15T03:06:02.136 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2024-09-15T03:06:02.221 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2024-09-15T03:06:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.231 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2024-09-15T03:06:02.231 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2024-09-15T03:06:02.297 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2024-09-15T03:06:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.304 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2024-09-15T03:06:02.304 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2024-09-15T03:06:02.353 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 26 20 28753 28733 8.63164 6.21875 0.00956017 0.01809 2024-09-15T03:06:02.393 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2024-09-15T03:06:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.408 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2024-09-15T03:06:02.408 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2024-09-15T03:06:02.518 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2024-09-15T03:06:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.533 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2024-09-15T03:06:02.533 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2024-09-15T03:06:02.637 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2024-09-15T03:06:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.643 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2024-09-15T03:06:02.643 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2024-09-15T03:06:02.773 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2024-09-15T03:06:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.781 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2024-09-15T03:06:02.782 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2024-09-15T03:06:02.889 INFO:teuthology.orchestra.run.smithi060.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2024-09-15T03:06:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:06:02.893 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T03:06:02.893 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T03:06:02.896 INFO:tasks.ceph.osd.1:Started 2024-09-15T03:06:02.896 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:06:03.092 INFO:teuthology.orchestra.run.smithi060.stderr:no valid command found; 10 closest matches: 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:0 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:1 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:2 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:abort 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:assert 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:config diff 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:config diff get 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:config get 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:config help [] 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:config set ... 2024-09-15T03:06:03.093 INFO:teuthology.orchestra.run.smithi060.stderr:admin_socket: invalid command 2024-09-15T03:06:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T03:06:03.095 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-15T03:06:03.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 27 20 29631 29611 8.56587 6.85938 0.0117936 0.0182307 2024-09-15T03:06:03.367 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:06:03.361+0000 7f08f8e31540 -1 Falling back to public interface 2024-09-15T03:06:04.157 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:06:04.149+0000 7f08f8e31540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:06:04.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 28 20 30518 30498 8.50742 6.92969 0.0174711 0.0183521 2024-09-15T03:06:05.179 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:06:05.173+0000 7f08f8e31540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-15T03:06:05.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 29 20 31366 31346 8.4425 6.625 0.0159958 0.0184951 2024-09-15T03:06:05.654 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:06:05.649+0000 7f08f8e31540 -1 osd.1 19 log_to_monitors true 2024-09-15T03:06:05.845 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-09-15T03:06:05.833+0000 7f08f137c700 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:06:06.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 30 20 32229 32209 8.38582 6.74219 0.0178705 0.0186217 2024-09-15T03:06:07.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 31 20 32638 32618 8.21838 3.19531 0.0244334 0.0186824 2024-09-15T03:06:08.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:06:08.203 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:06:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "ops": [], 2024-09-15T03:06:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "num_ops": 0 2024-09-15T03:06:08.203 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:06:08.212 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2024-09-15T03:06:08.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:06:08.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 32 20 32638 32618 7.96159 0 - 0.0186824 2024-09-15T03:06:08.495 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:08.496 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":30,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:06:07.826015+0000","last_up_change":"2024-09-15T03:06:06.823776+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:04:52.771663+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":29,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":28132},{"type":"v1","addr":"172.21.15.60:6801","nonce":28132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":28132},{"type":"v1","addr":"172.21.15.60:6803","nonce":28132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":28132},{"type":"v1","addr":"172.21.15.60:6807","nonce":28132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":28132},{"type":"v1","addr":"172.21.15.60:6805","nonce":28132}]},"public_addr":"172.21.15.60:6801/28132","cluster_addr":"172.21.15.60:6803/28132","heartbeat_back_addr":"172.21.15.60:6807/28132","heartbeat_front_addr":"172.21.15.60:6805/28132","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":29,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":32304},{"type":"v1","addr":"172.21.15.60:6817","nonce":32304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":32304},{"type":"v1","addr":"172.21.15.60:6819","nonce":32304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":32304},{"type":"v1","addr":"172.21.15.60:6823","nonce":32304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":32304},{"type":"v1","addr":"172.21.15.60:6821","nonce":32304}]},"public_addr":"172.21.15.60:6817/32304","cluster_addr":"172.21.15.60:6819/32304","heartbeat_back_addr":"172.21.15.60:6823/32304","heartbeat_front_addr":"172.21.15.60:6821/32304","state":["exists","up"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":29,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":29026},{"type":"v1","addr":"172.21.15.60:6809","nonce":29026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":29026},{"type":"v1","addr":"172.21.15.60:6811","nonce":29026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":29026},{"type":"v1","addr":"172.21.15.60:6815","nonce":29026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":29026},{"type":"v1","addr":"172.21.15.60:6813","nonce":29026}]},"public_addr":"172.21.15.60:6809/29026","cluster_addr":"172.21.15.60:6811/29026","heartbeat_back_addr":"172.21.15.60:6815/29026","heartbeat_front_addr":"172.21.15.60:6813/29026","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[2147483647,0,2,3]},{"pgid":"1.1","osds":[3,2147483647,0,2]},{"pgid":"1.2","osds":[0,3,2147483647,2]},{"pgid":"1.3","osds":[2147483647,0,2,3]},{"pgid":"1.4","osds":[3,0,2,2147483647]},{"pgid":"1.5","osds":[2,0,3,2147483647]},{"pgid":"1.6","osds":[3,0,2147483647,2]},{"pgid":"1.7","osds":[2147483647,3,2,0]},{"pgid":"1.8","osds":[2147483647,2,3,0]},{"pgid":"1.9","osds":[2147483647,3,2,0]},{"pgid":"1.a","osds":[2147483647,2,3,0]},{"pgid":"1.b","osds":[2147483647,0,2,3]},{"pgid":"1.c","osds":[3,0,2,2147483647]},{"pgid":"1.d","osds":[2,2147483647,3,0]},{"pgid":"1.e","osds":[2147483647,0,3,2]},{"pgid":"1.f","osds":[3,0,2147483647,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:06:08.513 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2024-09-15T03:06:08.513 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-15T03:06:08.513 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:08.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:08.800 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:08.810 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-09-15T03:06:08.648099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":260053564,"num_objects":31749,"num_object_clones":0,"num_object_copies":126996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31776,"num_write_kb":254090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31803,"ondisk_log_size":31803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1578996,"kb_used_data":379792,"kb_used_omap":14,"kb_used_meta":1199153,"kb_avail":375908364,"statfs":{"total":386547056640,"available":384930164736,"internally_reserved":0,"allocated":388907008,"data_stored":388365886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227932699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,5,8,8,10,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35176448,"num_objects":4294,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4294,"num_write_kb":34352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001338"},"pg_stats":[{"pgid":"1.f","version":"27'1892","reported_seq":1921,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.086413+0000","last_change":"2024-09-15T03:05:36.363087+0000","last_active":"2024-09-15T03:06:05.086413+0000","last_peered":"2024-09-15T03:06:05.086413+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.078107+0000","last_became_peered":"2024-09-15T03:05:27.078107+0000","last_unstale":"2024-09-15T03:06:05.086413+0000","last_undegraded":"2024-09-15T03:05:36.359817+0000","last_fullsized":"2024-09-15T03:05:27.059184+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1892,"log_dups_size":0,"ondisk_log_size":1892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:26.729372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15499264,"num_objects":1892,"num_object_clones":0,"num_object_copies":7568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1892,"num_write_kb":15136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1957","reported_seq":2008,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.850911+0000","last_change":"2024-09-15T03:05:47.437364+0000","last_active":"2024-09-15T03:06:05.850911+0000","last_peered":"2024-09-15T03:06:05.850911+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:05:27.076040+0000","last_became_peered":"2024-09-15T03:05:27.076040+0000","last_unstale":"2024-09-15T03:06:05.850911+0000","last_undegraded":"2024-09-15T03:05:27.063502+0000","last_fullsized":"2024-09-15T03:05:27.062931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1957,"log_dups_size":0,"ondisk_log_size":1957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15996530,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1955,"num_write_kb":15636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1953}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2100","reported_seq":2166,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.655673+0000","last_change":"2024-09-15T03:05:47.986686+0000","last_active":"2024-09-15T03:06:06.655673+0000","last_peered":"2024-09-15T03:06:06.655673+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:05:27.073076+0000","last_became_peered":"2024-09-15T03:05:27.073076+0000","last_unstale":"2024-09-15T03:06:06.655673+0000","last_undegraded":"2024-09-15T03:05:27.062455+0000","last_fullsized":"2024-09-15T03:05:27.061642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17177401,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":16782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2097}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1913","reported_seq":1964,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.965148+0000","last_change":"2024-09-15T03:05:48.701594+0000","last_active":"2024-09-15T03:06:05.965148+0000","last_peered":"2024-09-15T03:06:05.965148+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:05:27.077226+0000","last_became_peered":"2024-09-15T03:05:27.077226+0000","last_unstale":"2024-09-15T03:06:05.965148+0000","last_undegraded":"2024-09-15T03:05:27.064312+0000","last_fullsized":"2024-09-15T03:05:27.063909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1913,"log_dups_size":0,"ondisk_log_size":1913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15636082,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1911,"num_write_kb":15277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1909}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'2021","reported_seq":2061,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.104748+0000","last_change":"2024-09-15T03:05:49.245426+0000","last_active":"2024-09-15T03:06:05.104748+0000","last_peered":"2024-09-15T03:06:05.104748+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:05:27.070138+0000","last_became_peered":"2024-09-15T03:05:27.070138+0000","last_unstale":"2024-09-15T03:06:05.104748+0000","last_undegraded":"2024-09-15T03:05:27.057975+0000","last_fullsized":"2024-09-15T03:05:27.057143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16529010,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":16149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1963","reported_seq":1991,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.070481+0000","last_change":"2024-09-15T03:05:36.377095+0000","last_active":"2024-09-15T03:06:05.070481+0000","last_peered":"2024-09-15T03:06:05.070481+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.069510+0000","last_became_peered":"2024-09-15T03:05:27.069510+0000","last_unstale":"2024-09-15T03:06:05.070481+0000","last_undegraded":"2024-09-15T03:05:36.359227+0000","last_fullsized":"2024-09-15T03:05:27.057867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1963,"log_dups_size":0,"ondisk_log_size":1963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:56.697498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":1963,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1963,"num_write_kb":15704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2012","reported_seq":2063,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.990659+0000","last_change":"2024-09-15T03:05:49.758375+0000","last_active":"2024-09-15T03:06:05.990659+0000","last_peered":"2024-09-15T03:06:05.990659+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:05:27.073004+0000","last_became_peered":"2024-09-15T03:05:27.073004+0000","last_unstale":"2024-09-15T03:06:05.990659+0000","last_undegraded":"2024-09-15T03:05:27.062946+0000","last_fullsized":"2024-09-15T03:05:27.061959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2012,"log_dups_size":0,"ondisk_log_size":2012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430706,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":16060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2006}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2029","reported_seq":2080,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.992089+0000","last_change":"2024-09-15T03:05:50.289840+0000","last_active":"2024-09-15T03:06:05.992089+0000","last_peered":"2024-09-15T03:06:05.992089+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:05:27.076435+0000","last_became_peered":"2024-09-15T03:05:27.076435+0000","last_unstale":"2024-09-15T03:06:05.992089+0000","last_undegraded":"2024-09-15T03:05:27.064600+0000","last_fullsized":"2024-09-15T03:05:27.063343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16578162,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1943","reported_seq":1983,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.105725+0000","last_change":"2024-09-15T03:05:50.981389+0000","last_active":"2024-09-15T03:06:05.105725+0000","last_peered":"2024-09-15T03:06:05.105725+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:05:27.079178+0000","last_became_peered":"2024-09-15T03:05:27.079178+0000","last_unstale":"2024-09-15T03:06:05.105725+0000","last_undegraded":"2024-09-15T03:05:27.058589+0000","last_fullsized":"2024-09-15T03:05:27.057594+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1943,"log_dups_size":0,"ondisk_log_size":1943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15846628,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1940,"num_write_kb":15497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2058","reported_seq":2126,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.656504+0000","last_change":"2024-09-15T03:05:51.509375+0000","last_active":"2024-09-15T03:06:06.656504+0000","last_peered":"2024-09-15T03:06:06.656504+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:05:27.073080+0000","last_became_peered":"2024-09-15T03:05:27.073080+0000","last_unstale":"2024-09-15T03:06:06.656504+0000","last_undegraded":"2024-09-15T03:05:27.060756+0000","last_fullsized":"2024-09-15T03:05:27.059796+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16840306,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2056}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1984","reported_seq":2025,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.085183+0000","last_change":"2024-09-15T03:05:52.241020+0000","last_active":"2024-09-15T03:06:05.085183+0000","last_peered":"2024-09-15T03:06:05.085183+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:05:27.069721+0000","last_became_peered":"2024-09-15T03:05:27.069721+0000","last_unstale":"2024-09-15T03:06:05.085183+0000","last_undegraded":"2024-09-15T03:05:27.058374+0000","last_fullsized":"2024-09-15T03:05:27.057414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1984,"log_dups_size":0,"ondisk_log_size":1984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16208299,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":15836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1899","reported_seq":1924,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.081635+0000","last_change":"2024-09-15T03:05:52.701122+0000","last_active":"2024-09-15T03:06:05.081635+0000","last_peered":"2024-09-15T03:06:05.081635+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:05:27.070691+0000","last_became_peered":"2024-09-15T03:05:27.070691+0000","last_unstale":"2024-09-15T03:06:05.081635+0000","last_undegraded":"2024-09-15T03:05:27.064197+0000","last_fullsized":"2024-09-15T03:05:27.063869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1899,"log_dups_size":0,"ondisk_log_size":1899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15547193,"num_objects":1898,"num_object_clones":0,"num_object_copies":7592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1898,"num_write_kb":15183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2039","reported_seq":2104,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.709569+0000","last_change":"2024-09-15T03:05:53.237197+0000","last_active":"2024-09-15T03:06:06.709569+0000","last_peered":"2024-09-15T03:06:06.709569+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:05:27.073296+0000","last_became_peered":"2024-09-15T03:05:27.073296+0000","last_unstale":"2024-09-15T03:06:06.709569+0000","last_undegraded":"2024-09-15T03:05:27.065069+0000","last_fullsized":"2024-09-15T03:05:27.064221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2039,"log_dups_size":0,"ondisk_log_size":2039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16694073,"num_objects":2038,"num_object_clones":0,"num_object_copies":8152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":16303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1965","reported_seq":2002,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.072476+0000","last_change":"2024-09-15T03:05:53.844392+0000","last_active":"2024-09-15T03:06:05.072476+0000","last_peered":"2024-09-15T03:06:05.072476+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:05:27.071179+0000","last_became_peered":"2024-09-15T03:05:27.071179+0000","last_unstale":"2024-09-15T03:06:05.072476+0000","last_undegraded":"2024-09-15T03:05:27.059665+0000","last_fullsized":"2024-09-15T03:05:27.057178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1965,"log_dups_size":0,"ondisk_log_size":1965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16062066,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1963,"num_write_kb":15700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1988","reported_seq":2057,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.693068+0000","last_change":"2024-09-15T03:05:54.363436+0000","last_active":"2024-09-15T03:06:06.693068+0000","last_peered":"2024-09-15T03:06:06.693068+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:05:27.074164+0000","last_became_peered":"2024-09-15T03:05:27.074164+0000","last_unstale":"2024-09-15T03:06:06.693068+0000","last_undegraded":"2024-09-15T03:05:27.063361+0000","last_fullsized":"2024-09-15T03:05:27.062449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1988,"log_dups_size":0,"ondisk_log_size":1988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16258674,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1986,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'2040","reported_seq":2090,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.009118+0000","last_change":"2024-09-15T03:05:54.735419+0000","last_active":"2024-09-15T03:06:06.009118+0000","last_peered":"2024-09-15T03:06:06.009118+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:05:27.076127+0000","last_became_peered":"2024-09-15T03:05:27.076127+0000","last_unstale":"2024-09-15T03:06:06.009118+0000","last_undegraded":"2024-09-15T03:05:27.063025+0000","last_fullsized":"2024-09-15T03:05:27.061985+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16668274,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":16292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2035}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":260053564,"num_objects":31749,"num_object_clones":0,"num_object_copies":126996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31776,"num_write_kb":254090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":391790592,"data_stored":391790592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31803,"ondisk_log_size":31803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422660,"kb_used_data":124796,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93949180,"statfs":{"total":96636764160,"available":96203960320,"internally_reserved":0,"allocated":127791104,"data_stored":127647515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431556,"kb_used_data":130040,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93940284,"statfs":{"total":96636764160,"available":96194850816,"internally_reserved":0,"allocated":133160960,"data_stored":133016613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298136,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073704,"statfs":{"total":96636764160,"available":96331472896,"internally_reserved":0,"allocated":278528,"data_stored":168931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426644,"kb_used_data":124684,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945196,"statfs":{"total":96636764160,"available":96199880704,"internally_reserved":0,"allocated":127676416,"data_stored":127532827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130748416,"data_stored":130748416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133328896,"data_stored":133328896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127598592,"data_stored":127598592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:08.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:09.234 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:09.237 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:09.247 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-15T03:06:08.832054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":260053564,"num_objects":31749,"num_object_clones":0,"num_object_copies":126996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31776,"num_write_kb":254090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31803,"ondisk_log_size":31803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1578996,"kb_used_data":379792,"kb_used_omap":14,"kb_used_meta":1199153,"kb_avail":375908364,"statfs":{"total":386547056640,"available":384930164736,"internally_reserved":0,"allocated":388907008,"data_stored":388365886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227932699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,5,8,8,10,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35176448,"num_objects":4294,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4294,"num_write_kb":34352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.184838"},"pg_stats":[{"pgid":"1.f","version":"27'1892","reported_seq":1921,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.086413+0000","last_change":"2024-09-15T03:05:36.363087+0000","last_active":"2024-09-15T03:06:05.086413+0000","last_peered":"2024-09-15T03:06:05.086413+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.078107+0000","last_became_peered":"2024-09-15T03:05:27.078107+0000","last_unstale":"2024-09-15T03:06:05.086413+0000","last_undegraded":"2024-09-15T03:05:36.359817+0000","last_fullsized":"2024-09-15T03:05:27.059184+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1892,"log_dups_size":0,"ondisk_log_size":1892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:26.729372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15499264,"num_objects":1892,"num_object_clones":0,"num_object_copies":7568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1892,"num_write_kb":15136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1957","reported_seq":2008,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.850911+0000","last_change":"2024-09-15T03:05:47.437364+0000","last_active":"2024-09-15T03:06:05.850911+0000","last_peered":"2024-09-15T03:06:05.850911+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:05:27.076040+0000","last_became_peered":"2024-09-15T03:05:27.076040+0000","last_unstale":"2024-09-15T03:06:05.850911+0000","last_undegraded":"2024-09-15T03:05:27.063502+0000","last_fullsized":"2024-09-15T03:05:27.062931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1957,"log_dups_size":0,"ondisk_log_size":1957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15996530,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1955,"num_write_kb":15636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1953}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2100","reported_seq":2166,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.655673+0000","last_change":"2024-09-15T03:05:47.986686+0000","last_active":"2024-09-15T03:06:06.655673+0000","last_peered":"2024-09-15T03:06:06.655673+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:05:27.073076+0000","last_became_peered":"2024-09-15T03:05:27.073076+0000","last_unstale":"2024-09-15T03:06:06.655673+0000","last_undegraded":"2024-09-15T03:05:27.062455+0000","last_fullsized":"2024-09-15T03:05:27.061642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17177401,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":16782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2097}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1913","reported_seq":1964,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.965148+0000","last_change":"2024-09-15T03:05:48.701594+0000","last_active":"2024-09-15T03:06:05.965148+0000","last_peered":"2024-09-15T03:06:05.965148+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:05:27.077226+0000","last_became_peered":"2024-09-15T03:05:27.077226+0000","last_unstale":"2024-09-15T03:06:05.965148+0000","last_undegraded":"2024-09-15T03:05:27.064312+0000","last_fullsized":"2024-09-15T03:05:27.063909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1913,"log_dups_size":0,"ondisk_log_size":1913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15636082,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1911,"num_write_kb":15277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1909}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'2021","reported_seq":2061,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.104748+0000","last_change":"2024-09-15T03:05:49.245426+0000","last_active":"2024-09-15T03:06:05.104748+0000","last_peered":"2024-09-15T03:06:05.104748+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:05:27.070138+0000","last_became_peered":"2024-09-15T03:05:27.070138+0000","last_unstale":"2024-09-15T03:06:05.104748+0000","last_undegraded":"2024-09-15T03:05:27.057975+0000","last_fullsized":"2024-09-15T03:05:27.057143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16529010,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":16149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1963","reported_seq":1991,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.070481+0000","last_change":"2024-09-15T03:05:36.377095+0000","last_active":"2024-09-15T03:06:05.070481+0000","last_peered":"2024-09-15T03:06:05.070481+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.069510+0000","last_became_peered":"2024-09-15T03:05:27.069510+0000","last_unstale":"2024-09-15T03:06:05.070481+0000","last_undegraded":"2024-09-15T03:05:36.359227+0000","last_fullsized":"2024-09-15T03:05:27.057867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1963,"log_dups_size":0,"ondisk_log_size":1963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:56.697498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":1963,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1963,"num_write_kb":15704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2012","reported_seq":2063,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.990659+0000","last_change":"2024-09-15T03:05:49.758375+0000","last_active":"2024-09-15T03:06:05.990659+0000","last_peered":"2024-09-15T03:06:05.990659+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:05:27.073004+0000","last_became_peered":"2024-09-15T03:05:27.073004+0000","last_unstale":"2024-09-15T03:06:05.990659+0000","last_undegraded":"2024-09-15T03:05:27.062946+0000","last_fullsized":"2024-09-15T03:05:27.061959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2012,"log_dups_size":0,"ondisk_log_size":2012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430706,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":16060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2006}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2029","reported_seq":2080,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.992089+0000","last_change":"2024-09-15T03:05:50.289840+0000","last_active":"2024-09-15T03:06:05.992089+0000","last_peered":"2024-09-15T03:06:05.992089+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:05:27.076435+0000","last_became_peered":"2024-09-15T03:05:27.076435+0000","last_unstale":"2024-09-15T03:06:05.992089+0000","last_undegraded":"2024-09-15T03:05:27.064600+0000","last_fullsized":"2024-09-15T03:05:27.063343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16578162,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1943","reported_seq":1983,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.105725+0000","last_change":"2024-09-15T03:05:50.981389+0000","last_active":"2024-09-15T03:06:05.105725+0000","last_peered":"2024-09-15T03:06:05.105725+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:05:27.079178+0000","last_became_peered":"2024-09-15T03:05:27.079178+0000","last_unstale":"2024-09-15T03:06:05.105725+0000","last_undegraded":"2024-09-15T03:05:27.058589+0000","last_fullsized":"2024-09-15T03:05:27.057594+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1943,"log_dups_size":0,"ondisk_log_size":1943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15846628,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1940,"num_write_kb":15497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2058","reported_seq":2126,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.656504+0000","last_change":"2024-09-15T03:05:51.509375+0000","last_active":"2024-09-15T03:06:06.656504+0000","last_peered":"2024-09-15T03:06:06.656504+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:05:27.073080+0000","last_became_peered":"2024-09-15T03:05:27.073080+0000","last_unstale":"2024-09-15T03:06:06.656504+0000","last_undegraded":"2024-09-15T03:05:27.060756+0000","last_fullsized":"2024-09-15T03:05:27.059796+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16840306,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2056}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1984","reported_seq":2025,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.085183+0000","last_change":"2024-09-15T03:05:52.241020+0000","last_active":"2024-09-15T03:06:05.085183+0000","last_peered":"2024-09-15T03:06:05.085183+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:05:27.069721+0000","last_became_peered":"2024-09-15T03:05:27.069721+0000","last_unstale":"2024-09-15T03:06:05.085183+0000","last_undegraded":"2024-09-15T03:05:27.058374+0000","last_fullsized":"2024-09-15T03:05:27.057414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1984,"log_dups_size":0,"ondisk_log_size":1984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16208299,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":15836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1899","reported_seq":1924,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.081635+0000","last_change":"2024-09-15T03:05:52.701122+0000","last_active":"2024-09-15T03:06:05.081635+0000","last_peered":"2024-09-15T03:06:05.081635+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:05:27.070691+0000","last_became_peered":"2024-09-15T03:05:27.070691+0000","last_unstale":"2024-09-15T03:06:05.081635+0000","last_undegraded":"2024-09-15T03:05:27.064197+0000","last_fullsized":"2024-09-15T03:05:27.063869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1899,"log_dups_size":0,"ondisk_log_size":1899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15547193,"num_objects":1898,"num_object_clones":0,"num_object_copies":7592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1898,"num_write_kb":15183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2039","reported_seq":2104,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.709569+0000","last_change":"2024-09-15T03:05:53.237197+0000","last_active":"2024-09-15T03:06:06.709569+0000","last_peered":"2024-09-15T03:06:06.709569+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:05:27.073296+0000","last_became_peered":"2024-09-15T03:05:27.073296+0000","last_unstale":"2024-09-15T03:06:06.709569+0000","last_undegraded":"2024-09-15T03:05:27.065069+0000","last_fullsized":"2024-09-15T03:05:27.064221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2039,"log_dups_size":0,"ondisk_log_size":2039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16694073,"num_objects":2038,"num_object_clones":0,"num_object_copies":8152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":16303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1965","reported_seq":2002,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.072476+0000","last_change":"2024-09-15T03:05:53.844392+0000","last_active":"2024-09-15T03:06:05.072476+0000","last_peered":"2024-09-15T03:06:05.072476+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:05:27.071179+0000","last_became_peered":"2024-09-15T03:05:27.071179+0000","last_unstale":"2024-09-15T03:06:05.072476+0000","last_undegraded":"2024-09-15T03:05:27.059665+0000","last_fullsized":"2024-09-15T03:05:27.057178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1965,"log_dups_size":0,"ondisk_log_size":1965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16062066,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1963,"num_write_kb":15700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1988","reported_seq":2057,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.693068+0000","last_change":"2024-09-15T03:05:54.363436+0000","last_active":"2024-09-15T03:06:06.693068+0000","last_peered":"2024-09-15T03:06:06.693068+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:05:27.074164+0000","last_became_peered":"2024-09-15T03:05:27.074164+0000","last_unstale":"2024-09-15T03:06:06.693068+0000","last_undegraded":"2024-09-15T03:05:27.063361+0000","last_fullsized":"2024-09-15T03:05:27.062449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1988,"log_dups_size":0,"ondisk_log_size":1988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16258674,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1986,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'2040","reported_seq":2090,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.009118+0000","last_change":"2024-09-15T03:05:54.735419+0000","last_active":"2024-09-15T03:06:06.009118+0000","last_peered":"2024-09-15T03:06:06.009118+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:05:27.076127+0000","last_became_peered":"2024-09-15T03:05:27.076127+0000","last_unstale":"2024-09-15T03:06:06.009118+0000","last_undegraded":"2024-09-15T03:05:27.063025+0000","last_fullsized":"2024-09-15T03:05:27.061985+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16668274,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":16292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2035}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":260053564,"num_objects":31749,"num_object_clones":0,"num_object_copies":126996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31776,"num_write_kb":254090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":391790592,"data_stored":391790592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31803,"ondisk_log_size":31803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422660,"kb_used_data":124796,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93949180,"statfs":{"total":96636764160,"available":96203960320,"internally_reserved":0,"allocated":127791104,"data_stored":127647515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431556,"kb_used_data":130040,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93940284,"statfs":{"total":96636764160,"available":96194850816,"internally_reserved":0,"allocated":133160960,"data_stored":133016613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298136,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073704,"statfs":{"total":96636764160,"available":96331472896,"internally_reserved":0,"allocated":278528,"data_stored":168931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426644,"kb_used_data":124684,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945196,"statfs":{"total":96636764160,"available":96199880704,"internally_reserved":0,"allocated":127676416,"data_stored":127532827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130748416,"data_stored":130748416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133328896,"data_stored":133328896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127598592,"data_stored":127598592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:09.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:09.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 33 20 32679 32659 7.73007 0.160156 0.0232821 0.0201869 2024-09-15T03:06:10.251 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:10.252 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:10.279 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-15T03:06:08.832054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":260053564,"num_objects":31749,"num_object_clones":0,"num_object_copies":126996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31776,"num_write_kb":254090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31803,"ondisk_log_size":31803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1578996,"kb_used_data":379792,"kb_used_omap":14,"kb_used_meta":1199153,"kb_avail":375908364,"statfs":{"total":386547056640,"available":384930164736,"internally_reserved":0,"allocated":388907008,"data_stored":388365886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227932699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,5,8,8,10,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35176448,"num_objects":4294,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4294,"num_write_kb":34352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.184838"},"pg_stats":[{"pgid":"1.f","version":"27'1892","reported_seq":1921,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.086413+0000","last_change":"2024-09-15T03:05:36.363087+0000","last_active":"2024-09-15T03:06:05.086413+0000","last_peered":"2024-09-15T03:06:05.086413+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.078107+0000","last_became_peered":"2024-09-15T03:05:27.078107+0000","last_unstale":"2024-09-15T03:06:05.086413+0000","last_undegraded":"2024-09-15T03:05:36.359817+0000","last_fullsized":"2024-09-15T03:05:27.059184+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1892,"log_dups_size":0,"ondisk_log_size":1892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:37:26.729372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15499264,"num_objects":1892,"num_object_clones":0,"num_object_copies":7568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1892,"num_write_kb":15136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1957","reported_seq":2008,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.850911+0000","last_change":"2024-09-15T03:05:47.437364+0000","last_active":"2024-09-15T03:06:05.850911+0000","last_peered":"2024-09-15T03:06:05.850911+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:05:27.076040+0000","last_became_peered":"2024-09-15T03:05:27.076040+0000","last_unstale":"2024-09-15T03:06:05.850911+0000","last_undegraded":"2024-09-15T03:05:27.063502+0000","last_fullsized":"2024-09-15T03:05:27.062931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1957,"log_dups_size":0,"ondisk_log_size":1957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15996530,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1955,"num_write_kb":15636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1953}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2100","reported_seq":2166,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.655673+0000","last_change":"2024-09-15T03:05:47.986686+0000","last_active":"2024-09-15T03:06:06.655673+0000","last_peered":"2024-09-15T03:06:06.655673+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:05:27.073076+0000","last_became_peered":"2024-09-15T03:05:27.073076+0000","last_unstale":"2024-09-15T03:06:06.655673+0000","last_undegraded":"2024-09-15T03:05:27.062455+0000","last_fullsized":"2024-09-15T03:05:27.061642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17177401,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":16782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2097}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1913","reported_seq":1964,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.965148+0000","last_change":"2024-09-15T03:05:48.701594+0000","last_active":"2024-09-15T03:06:05.965148+0000","last_peered":"2024-09-15T03:06:05.965148+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:05:27.077226+0000","last_became_peered":"2024-09-15T03:05:27.077226+0000","last_unstale":"2024-09-15T03:06:05.965148+0000","last_undegraded":"2024-09-15T03:05:27.064312+0000","last_fullsized":"2024-09-15T03:05:27.063909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1913,"log_dups_size":0,"ondisk_log_size":1913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15636082,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1911,"num_write_kb":15277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1909}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'2021","reported_seq":2061,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.104748+0000","last_change":"2024-09-15T03:05:49.245426+0000","last_active":"2024-09-15T03:06:05.104748+0000","last_peered":"2024-09-15T03:06:05.104748+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:05:27.070138+0000","last_became_peered":"2024-09-15T03:05:27.070138+0000","last_unstale":"2024-09-15T03:06:05.104748+0000","last_undegraded":"2024-09-15T03:05:27.057975+0000","last_fullsized":"2024-09-15T03:05:27.057143+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2021,"log_dups_size":0,"ondisk_log_size":2021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16529010,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":16149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1963","reported_seq":1991,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.070481+0000","last_change":"2024-09-15T03:05:36.377095+0000","last_active":"2024-09-15T03:06:05.070481+0000","last_peered":"2024-09-15T03:06:05.070481+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:05:27.069510+0000","last_became_peered":"2024-09-15T03:05:27.069510+0000","last_unstale":"2024-09-15T03:06:05.070481+0000","last_undegraded":"2024-09-15T03:05:36.359227+0000","last_fullsized":"2024-09-15T03:05:27.057867+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1963,"log_dups_size":0,"ondisk_log_size":1963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:51:56.697498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":1963,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1963,"num_write_kb":15704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2012","reported_seq":2063,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.990659+0000","last_change":"2024-09-15T03:05:49.758375+0000","last_active":"2024-09-15T03:06:05.990659+0000","last_peered":"2024-09-15T03:06:05.990659+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:05:27.073004+0000","last_became_peered":"2024-09-15T03:05:27.073004+0000","last_unstale":"2024-09-15T03:06:05.990659+0000","last_undegraded":"2024-09-15T03:05:27.062946+0000","last_fullsized":"2024-09-15T03:05:27.061959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2012,"log_dups_size":0,"ondisk_log_size":2012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430706,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":16060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2006}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2029","reported_seq":2080,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.992089+0000","last_change":"2024-09-15T03:05:50.289840+0000","last_active":"2024-09-15T03:06:05.992089+0000","last_peered":"2024-09-15T03:06:05.992089+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:05:27.076435+0000","last_became_peered":"2024-09-15T03:05:27.076435+0000","last_unstale":"2024-09-15T03:06:05.992089+0000","last_undegraded":"2024-09-15T03:05:27.064600+0000","last_fullsized":"2024-09-15T03:05:27.063343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16578162,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1943","reported_seq":1983,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.105725+0000","last_change":"2024-09-15T03:05:50.981389+0000","last_active":"2024-09-15T03:06:05.105725+0000","last_peered":"2024-09-15T03:06:05.105725+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:05:27.079178+0000","last_became_peered":"2024-09-15T03:05:27.079178+0000","last_unstale":"2024-09-15T03:06:05.105725+0000","last_undegraded":"2024-09-15T03:05:27.058589+0000","last_fullsized":"2024-09-15T03:05:27.057594+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1943,"log_dups_size":0,"ondisk_log_size":1943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15846628,"num_objects":1935,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1940,"num_write_kb":15497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2058","reported_seq":2126,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.656504+0000","last_change":"2024-09-15T03:05:51.509375+0000","last_active":"2024-09-15T03:06:06.656504+0000","last_peered":"2024-09-15T03:06:06.656504+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:05:27.073080+0000","last_became_peered":"2024-09-15T03:05:27.073080+0000","last_unstale":"2024-09-15T03:06:06.656504+0000","last_undegraded":"2024-09-15T03:05:27.060756+0000","last_fullsized":"2024-09-15T03:05:27.059796+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16840306,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2056}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1984","reported_seq":2025,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.085183+0000","last_change":"2024-09-15T03:05:52.241020+0000","last_active":"2024-09-15T03:06:05.085183+0000","last_peered":"2024-09-15T03:06:05.085183+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:05:27.069721+0000","last_became_peered":"2024-09-15T03:05:27.069721+0000","last_unstale":"2024-09-15T03:06:05.085183+0000","last_undegraded":"2024-09-15T03:05:27.058374+0000","last_fullsized":"2024-09-15T03:05:27.057414+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1984,"log_dups_size":0,"ondisk_log_size":1984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16208299,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1981,"num_write_kb":15836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1899","reported_seq":1924,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.081635+0000","last_change":"2024-09-15T03:05:52.701122+0000","last_active":"2024-09-15T03:06:05.081635+0000","last_peered":"2024-09-15T03:06:05.081635+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:05:27.070691+0000","last_became_peered":"2024-09-15T03:05:27.070691+0000","last_unstale":"2024-09-15T03:06:05.081635+0000","last_undegraded":"2024-09-15T03:05:27.064197+0000","last_fullsized":"2024-09-15T03:05:27.063869+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1899,"log_dups_size":0,"ondisk_log_size":1899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15547193,"num_objects":1898,"num_object_clones":0,"num_object_copies":7592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1898,"num_write_kb":15183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2039","reported_seq":2104,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.709569+0000","last_change":"2024-09-15T03:05:53.237197+0000","last_active":"2024-09-15T03:06:06.709569+0000","last_peered":"2024-09-15T03:06:06.709569+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:05:27.073296+0000","last_became_peered":"2024-09-15T03:05:27.073296+0000","last_unstale":"2024-09-15T03:06:06.709569+0000","last_undegraded":"2024-09-15T03:05:27.065069+0000","last_fullsized":"2024-09-15T03:05:27.064221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2039,"log_dups_size":0,"ondisk_log_size":2039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16694073,"num_objects":2038,"num_object_clones":0,"num_object_copies":8152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":16303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1965","reported_seq":2002,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.072476+0000","last_change":"2024-09-15T03:05:53.844392+0000","last_active":"2024-09-15T03:06:05.072476+0000","last_peered":"2024-09-15T03:06:05.072476+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:05:27.071179+0000","last_became_peered":"2024-09-15T03:05:27.071179+0000","last_unstale":"2024-09-15T03:06:05.072476+0000","last_undegraded":"2024-09-15T03:05:27.059665+0000","last_fullsized":"2024-09-15T03:05:27.057178+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1965,"log_dups_size":0,"ondisk_log_size":1965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16062066,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1961,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1963,"num_write_kb":15700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1988","reported_seq":2057,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.693068+0000","last_change":"2024-09-15T03:05:54.363436+0000","last_active":"2024-09-15T03:06:06.693068+0000","last_peered":"2024-09-15T03:06:06.693068+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:05:27.074164+0000","last_became_peered":"2024-09-15T03:05:27.074164+0000","last_unstale":"2024-09-15T03:06:06.693068+0000","last_undegraded":"2024-09-15T03:05:27.063361+0000","last_fullsized":"2024-09-15T03:05:27.062449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1988,"log_dups_size":0,"ondisk_log_size":1988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16258674,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1986,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'2040","reported_seq":2090,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.009118+0000","last_change":"2024-09-15T03:05:54.735419+0000","last_active":"2024-09-15T03:06:06.009118+0000","last_peered":"2024-09-15T03:06:06.009118+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:05:27.076127+0000","last_became_peered":"2024-09-15T03:05:27.076127+0000","last_unstale":"2024-09-15T03:06:06.009118+0000","last_undegraded":"2024-09-15T03:05:27.063025+0000","last_fullsized":"2024-09-15T03:05:27.061985+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16668274,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":16292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2035}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":260053564,"num_objects":31749,"num_object_clones":0,"num_object_copies":126996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31749,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31776,"num_write_kb":254090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":391790592,"data_stored":391790592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31803,"ondisk_log_size":31803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422660,"kb_used_data":124796,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93949180,"statfs":{"total":96636764160,"available":96203960320,"internally_reserved":0,"allocated":127791104,"data_stored":127647515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,4,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431556,"kb_used_data":130040,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93940284,"statfs":{"total":96636764160,"available":96194850816,"internally_reserved":0,"allocated":133160960,"data_stored":133016613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298136,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073704,"statfs":{"total":96636764160,"available":96331472896,"internally_reserved":0,"allocated":278528,"data_stored":168931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426644,"kb_used_data":124684,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945196,"statfs":{"total":96636764160,"available":96199880704,"internally_reserved":0,"allocated":127676416,"data_stored":127532827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130748416,"data_stored":130748416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133328896,"data_stored":133328896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":127598592,"data_stored":127598592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:10.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:06:10.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "27'1892", 'reported_seq': 1921, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.086413+0000', 'last_change': '2024-09-15T03:05:36.363087+0000', 'last_active': '2024-09-15T03:06:05.086413+0000', 'last_peered': '2024-09-15T03:06:05.086413+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:05:27.078107+0000', 'last_became_peered': '2024-09-15T03:05:27.078107+0000', 'last_unstale': '2024-09-15T03:06:05.086413+0000', 'last_undegraded': '2024-09-15T03:05:36.359817+0000', 'last_fullsized': '2024-09-15T03:05:27.059184+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1892, 'log_dups_size': 0, 'ondisk_log_size': 1892, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T11:37:26.729372+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15499264, 'num_objects': 1892, 'num_object_clones': 0, 'num_object_copies': 7568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1892, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1892, 'num_write_kb': 15136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2147483647, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1892}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:10.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:06:10.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "28'1957", 'reported_seq': 2008, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.850911+0000', 'last_change': '2024-09-15T03:05:47.437364+0000', 'last_active': '2024-09-15T03:06:05.850911+0000', 'last_peered': '2024-09-15T03:06:05.850911+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:05:27.076040+0000', 'last_became_peered': '2024-09-15T03:05:27.076040+0000', 'last_unstale': '2024-09-15T03:06:05.850911+0000', 'last_undegraded': '2024-09-15T03:05:27.063502+0000', 'last_fullsized': '2024-09-15T03:05:27.062931+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1957, 'log_dups_size': 0, 'ondisk_log_size': 1957, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15996530, 'num_objects': 1953, 'num_object_clones': 0, 'num_object_copies': 7812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1955, 'num_write_kb': 15636, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1953}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:10.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:06:10.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "28'2100", 'reported_seq': 2166, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:06.655673+0000', 'last_change': '2024-09-15T03:05:47.986686+0000', 'last_active': '2024-09-15T03:06:06.655673+0000', 'last_peered': '2024-09-15T03:06:06.655673+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:05:27.073076+0000', 'last_became_peered': '2024-09-15T03:05:27.073076+0000', 'last_unstale': '2024-09-15T03:06:06.655673+0000', 'last_undegraded': '2024-09-15T03:05:27.062455+0000', 'last_fullsized': '2024-09-15T03:05:27.061642+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2100, 'log_dups_size': 0, 'ondisk_log_size': 2100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17177401, 'num_objects': 2097, 'num_object_clones': 0, 'num_object_copies': 8388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2097, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2099, 'num_write_kb': 16782, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 2147483647, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2097}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:10.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:06:10.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "28'1913", 'reported_seq': 1964, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.965148+0000', 'last_change': '2024-09-15T03:05:48.701594+0000', 'last_active': '2024-09-15T03:06:05.965148+0000', 'last_peered': '2024-09-15T03:06:05.965148+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:05:27.077226+0000', 'last_became_peered': '2024-09-15T03:05:27.077226+0000', 'last_unstale': '2024-09-15T03:06:05.965148+0000', 'last_undegraded': '2024-09-15T03:05:27.064312+0000', 'last_fullsized': '2024-09-15T03:05:27.063909+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1913, 'log_dups_size': 0, 'ondisk_log_size': 1913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15636082, 'num_objects': 1909, 'num_object_clones': 0, 'num_object_copies': 7636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1909, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1911, 'num_write_kb': 15277, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1909}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:10.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "27'2021", 'reported_seq': 2061, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.104748+0000', 'last_change': '2024-09-15T03:05:49.245426+0000', 'last_active': '2024-09-15T03:06:05.104748+0000', 'last_peered': '2024-09-15T03:06:05.104748+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:05:27.070138+0000', 'last_became_peered': '2024-09-15T03:05:27.070138+0000', 'last_unstale': '2024-09-15T03:06:05.104748+0000', 'last_undegraded': '2024-09-15T03:05:27.057975+0000', 'last_fullsized': '2024-09-15T03:05:27.057143+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2021, 'log_dups_size': 0, 'ondisk_log_size': 2021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16529010, 'num_objects': 2018, 'num_object_clones': 0, 'num_object_copies': 8072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2018, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2019, 'num_write_kb': 16149, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2147483647, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2018}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "27'1963", 'reported_seq': 1991, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.070481+0000', 'last_change': '2024-09-15T03:05:36.377095+0000', 'last_active': '2024-09-15T03:06:05.070481+0000', 'last_peered': '2024-09-15T03:06:05.070481+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:05:27.069510+0000', 'last_became_peered': '2024-09-15T03:05:27.069510+0000', 'last_unstale': '2024-09-15T03:06:05.070481+0000', 'last_undegraded': '2024-09-15T03:05:36.359227+0000', 'last_fullsized': '2024-09-15T03:05:27.057867+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1963, 'log_dups_size': 0, 'ondisk_log_size': 1963, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T04:51:56.697498+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16080896, 'num_objects': 1963, 'num_object_clones': 0, 'num_object_copies': 7852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1963, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1963, 'num_write_kb': 15704, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 2147483647], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1963}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "28'2012", 'reported_seq': 2063, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.990659+0000', 'last_change': '2024-09-15T03:05:49.758375+0000', 'last_active': '2024-09-15T03:06:05.990659+0000', 'last_peered': '2024-09-15T03:06:05.990659+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:05:27.073004+0000', 'last_became_peered': '2024-09-15T03:05:27.073004+0000', 'last_unstale': '2024-09-15T03:06:05.990659+0000', 'last_undegraded': '2024-09-15T03:05:27.062946+0000', 'last_fullsized': '2024-09-15T03:05:27.061959+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2012, 'log_dups_size': 0, 'ondisk_log_size': 2012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16430706, 'num_objects': 2006, 'num_object_clones': 0, 'num_object_copies': 8024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2010, 'num_write_kb': 16060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 2147483647, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2006}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "28'2029", 'reported_seq': 2080, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.992089+0000', 'last_change': '2024-09-15T03:05:50.289840+0000', 'last_active': '2024-09-15T03:06:05.992089+0000', 'last_peered': '2024-09-15T03:06:05.992089+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:05:27.076435+0000', 'last_became_peered': '2024-09-15T03:05:27.076435+0000', 'last_unstale': '2024-09-15T03:06:05.992089+0000', 'last_undegraded': '2024-09-15T03:05:27.064600+0000', 'last_fullsized': '2024-09-15T03:05:27.063343+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2029, 'log_dups_size': 0, 'ondisk_log_size': 2029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16578162, 'num_objects': 2024, 'num_object_clones': 0, 'num_object_copies': 8096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2024, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2027, 'num_write_kb': 16204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "27'1943", 'reported_seq': 1983, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.105725+0000', 'last_change': '2024-09-15T03:05:50.981389+0000', 'last_active': '2024-09-15T03:06:05.105725+0000', 'last_peered': '2024-09-15T03:06:05.105725+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:05:27.079178+0000', 'last_became_peered': '2024-09-15T03:05:27.079178+0000', 'last_unstale': '2024-09-15T03:06:05.105725+0000', 'last_undegraded': '2024-09-15T03:05:27.058589+0000', 'last_fullsized': '2024-09-15T03:05:27.057594+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1943, 'log_dups_size': 0, 'ondisk_log_size': 1943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15846628, 'num_objects': 1935, 'num_object_clones': 0, 'num_object_copies': 7740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1940, 'num_write_kb': 15497, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 2147483647], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:10.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "28'2058", 'reported_seq': 2126, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:06.656504+0000', 'last_change': '2024-09-15T03:05:51.509375+0000', 'last_active': '2024-09-15T03:06:06.656504+0000', 'last_peered': '2024-09-15T03:06:06.656504+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:05:27.073080+0000', 'last_became_peered': '2024-09-15T03:05:27.073080+0000', 'last_unstale': '2024-09-15T03:06:06.656504+0000', 'last_undegraded': '2024-09-15T03:05:27.060756+0000', 'last_fullsized': '2024-09-15T03:05:27.059796+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2058, 'log_dups_size': 0, 'ondisk_log_size': 2058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16840306, 'num_objects': 2056, 'num_object_clones': 0, 'num_object_copies': 8224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2056, 'num_write_kb': 16446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 2147483647], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2056}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "27'1984", 'reported_seq': 2025, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.085183+0000', 'last_change': '2024-09-15T03:05:52.241020+0000', 'last_active': '2024-09-15T03:06:05.085183+0000', 'last_peered': '2024-09-15T03:06:05.085183+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:05:27.069721+0000', 'last_became_peered': '2024-09-15T03:05:27.069721+0000', 'last_unstale': '2024-09-15T03:06:05.085183+0000', 'last_undegraded': '2024-09-15T03:05:27.058374+0000', 'last_fullsized': '2024-09-15T03:05:27.057414+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1984, 'log_dups_size': 0, 'ondisk_log_size': 1984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16208299, 'num_objects': 1979, 'num_object_clones': 0, 'num_object_copies': 7916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1981, 'num_write_kb': 15836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2147483647, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1979}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "27'1899", 'reported_seq': 1924, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.081635+0000', 'last_change': '2024-09-15T03:05:52.701122+0000', 'last_active': '2024-09-15T03:06:05.081635+0000', 'last_peered': '2024-09-15T03:06:05.081635+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:05:27.070691+0000', 'last_became_peered': '2024-09-15T03:05:27.070691+0000', 'last_unstale': '2024-09-15T03:06:05.081635+0000', 'last_undegraded': '2024-09-15T03:05:27.064197+0000', 'last_fullsized': '2024-09-15T03:05:27.063869+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1899, 'log_dups_size': 0, 'ondisk_log_size': 1899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15547193, 'num_objects': 1898, 'num_object_clones': 0, 'num_object_copies': 7592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1898, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1898, 'num_write_kb': 15183, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1898}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "28'2039", 'reported_seq': 2104, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:06.709569+0000', 'last_change': '2024-09-15T03:05:53.237197+0000', 'last_active': '2024-09-15T03:06:06.709569+0000', 'last_peered': '2024-09-15T03:06:06.709569+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:05:27.073296+0000', 'last_became_peered': '2024-09-15T03:05:27.073296+0000', 'last_unstale': '2024-09-15T03:06:06.709569+0000', 'last_undegraded': '2024-09-15T03:05:27.065069+0000', 'last_fullsized': '2024-09-15T03:05:27.064221+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2039, 'log_dups_size': 0, 'ondisk_log_size': 2039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16694073, 'num_objects': 2038, 'num_object_clones': 0, 'num_object_copies': 8152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2038, 'num_write_kb': 16303, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2038}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "27'1965", 'reported_seq': 2002, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:05.072476+0000', 'last_change': '2024-09-15T03:05:53.844392+0000', 'last_active': '2024-09-15T03:06:05.072476+0000', 'last_peered': '2024-09-15T03:06:05.072476+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:05:27.071179+0000', 'last_became_peered': '2024-09-15T03:05:27.071179+0000', 'last_unstale': '2024-09-15T03:06:05.072476+0000', 'last_undegraded': '2024-09-15T03:05:27.059665+0000', 'last_fullsized': '2024-09-15T03:05:27.057178+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1965, 'log_dups_size': 0, 'ondisk_log_size': 1965, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16062066, 'num_objects': 1961, 'num_object_clones': 0, 'num_object_copies': 7844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1961, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1963, 'num_write_kb': 15700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:06:10.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "28'1988", 'reported_seq': 2057, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:06.693068+0000', 'last_change': '2024-09-15T03:05:54.363436+0000', 'last_active': '2024-09-15T03:06:06.693068+0000', 'last_peered': '2024-09-15T03:06:06.693068+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:05:27.074164+0000', 'last_became_peered': '2024-09-15T03:05:27.074164+0000', 'last_unstale': '2024-09-15T03:06:06.693068+0000', 'last_undegraded': '2024-09-15T03:05:27.063361+0000', 'last_fullsized': '2024-09-15T03:05:27.062449+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 1988, 'log_dups_size': 0, 'ondisk_log_size': 1988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16258674, 'num_objects': 1985, 'num_object_clones': 0, 'num_object_copies': 7940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1986, 'num_write_kb': 15885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1985}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:10.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:06:10.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "28'2040", 'reported_seq': 2090, 'reported_epoch': 28, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T03:06:06.009118+0000', 'last_change': '2024-09-15T03:05:54.735419+0000', 'last_active': '2024-09-15T03:06:06.009118+0000', 'last_peered': '2024-09-15T03:06:06.009118+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:05:27.076127+0000', 'last_became_peered': '2024-09-15T03:05:27.076127+0000', 'last_unstale': '2024-09-15T03:06:06.009118+0000', 'last_undegraded': '2024-09-15T03:05:27.063025+0000', 'last_fullsized': '2024-09-15T03:05:27.061985+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2040, 'log_dups_size': 0, 'ondisk_log_size': 2040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16668274, 'num_objects': 2035, 'num_object_clones': 0, 'num_object_copies': 8140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2035, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2038, 'num_write_kb': 16292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2035}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:10.284 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:06:10.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 34 20 33089 33069 7.59693 3.20312 0.0548625 0.0205457 2024-09-15T03:06:11.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 35 20 33498 33478 7.47118 3.19531 0.027982 0.0208867 2024-09-15T03:06:11.460 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:11.460 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32395/130240 objects degraded (24.873%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":94,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32560,"data_bytes":266697276,"bytes_used":1632993280,"bytes_avail":384914063360,"bytes_total":386547056640,"degraded_objects":32395,"degraded_total":130240,"degraded_ratio":0.24873310810810811,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":8482,"recovering_keys_per_sec":0,"num_objects_recovered":5,"num_bytes_recovered":40960,"num_keys_recovered":0,"write_bytes_sec":5473230,"read_op_per_sec":0,"write_op_per_sec":668},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-15T03:05:48.642017+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T03:06:11.482 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:06:11.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:12.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 36 20 33901 33881 7.35111 3.14844 0.0837413 0.0212321 2024-09-15T03:06:12.518 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:12.527 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:12.553 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":70,"stamp":"2024-09-15T03:06:10.648783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":266697276,"num_objects":32560,"num_object_clones":0,"num_object_copies":130240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32587,"num_write_kb":260578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32614,"ondisk_log_size":32614,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1594720,"kb_used_data":387196,"kb_used_omap":14,"kb_used_meta":1207473,"kb_avail":375892640,"statfs":{"total":386547056640,"available":384914063360,"internally_reserved":0,"allocated":396488704,"data_stored":395932193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236452379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,5,5,4,4,1,0,0,4,0,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26427392,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3226,"num_write_kb":25808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.828481"},"pg_stats":[{"pgid":"1.f","version":"31'2002","reported_seq":2044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.109008+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:10.109008+0000","last_peered":"2024-09-15T03:06:10.109008+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:10.109008+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2002,"log_dups_size":0,"ondisk_log_size":2002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16400384,"num_objects":2002,"num_object_clones":0,"num_object_copies":8008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2002,"num_write_kb":16016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"28'1957","reported_seq":2008,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.850911+0000","last_change":"2024-09-15T03:05:47.437364+0000","last_active":"2024-09-15T03:06:05.850911+0000","last_peered":"2024-09-15T03:06:05.850911+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:05:27.076040+0000","last_became_peered":"2024-09-15T03:05:27.076040+0000","last_unstale":"2024-09-15T03:06:05.850911+0000","last_undegraded":"2024-09-15T03:05:27.063502+0000","last_fullsized":"2024-09-15T03:05:27.062931+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1957,"log_dups_size":0,"ondisk_log_size":1957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15996530,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1955,"num_write_kb":15636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1953}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"28'2100","reported_seq":2166,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.655673+0000","last_change":"2024-09-15T03:05:47.986686+0000","last_active":"2024-09-15T03:06:06.655673+0000","last_peered":"2024-09-15T03:06:06.655673+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:05:27.073076+0000","last_became_peered":"2024-09-15T03:05:27.073076+0000","last_unstale":"2024-09-15T03:06:06.655673+0000","last_undegraded":"2024-09-15T03:05:27.062455+0000","last_fullsized":"2024-09-15T03:05:27.061642+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17177401,"num_objects":2097,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":16782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2097}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"28'1913","reported_seq":1964,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.965148+0000","last_change":"2024-09-15T03:05:48.701594+0000","last_active":"2024-09-15T03:06:05.965148+0000","last_peered":"2024-09-15T03:06:05.965148+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:05:27.077226+0000","last_became_peered":"2024-09-15T03:05:27.077226+0000","last_unstale":"2024-09-15T03:06:05.965148+0000","last_undegraded":"2024-09-15T03:05:27.064312+0000","last_fullsized":"2024-09-15T03:05:27.063909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1913,"log_dups_size":0,"ondisk_log_size":1913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15636082,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1911,"num_write_kb":15277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1909}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2136","reported_seq":2192,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:09.992397+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:09.992397+0000","last_peered":"2024-09-15T03:06:09.992397+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:09.992397+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17471090,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2134,"num_write_kb":17069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2080","reported_seq":2124,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.084924+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:10.084924+0000","last_peered":"2024-09-15T03:06:10.084924+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:10.084924+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2080,"log_dups_size":0,"ondisk_log_size":2080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039360,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2080,"num_write_kb":16640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":24}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2012","reported_seq":2063,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.990659+0000","last_change":"2024-09-15T03:05:49.758375+0000","last_active":"2024-09-15T03:06:05.990659+0000","last_peered":"2024-09-15T03:06:05.990659+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:05:27.073004+0000","last_became_peered":"2024-09-15T03:05:27.073004+0000","last_unstale":"2024-09-15T03:06:05.990659+0000","last_undegraded":"2024-09-15T03:05:27.062946+0000","last_fullsized":"2024-09-15T03:05:27.061959+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2012,"log_dups_size":0,"ondisk_log_size":2012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430706,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":16060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2006}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"28'2029","reported_seq":2080,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:05.992089+0000","last_change":"2024-09-15T03:05:50.289840+0000","last_active":"2024-09-15T03:06:05.992089+0000","last_peered":"2024-09-15T03:06:05.992089+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:05:27.076435+0000","last_became_peered":"2024-09-15T03:05:27.076435+0000","last_unstale":"2024-09-15T03:06:05.992089+0000","last_undegraded":"2024-09-15T03:05:27.064600+0000","last_fullsized":"2024-09-15T03:05:27.063343+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16578162,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2071","reported_seq":2128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.042921+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:10.042921+0000","last_peered":"2024-09-15T03:06:10.042921+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:10.042921+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2071,"log_dups_size":0,"ondisk_log_size":2071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16895204,"num_objects":2063,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2068,"num_write_kb":16521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":21}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"28'2058","reported_seq":2126,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.656504+0000","last_change":"2024-09-15T03:05:51.509375+0000","last_active":"2024-09-15T03:06:06.656504+0000","last_peered":"2024-09-15T03:06:06.656504+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:05:27.073080+0000","last_became_peered":"2024-09-15T03:05:27.073080+0000","last_unstale":"2024-09-15T03:06:06.656504+0000","last_undegraded":"2024-09-15T03:05:27.060756+0000","last_fullsized":"2024-09-15T03:05:27.059796+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2058,"log_dups_size":0,"ondisk_log_size":2058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16840306,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":16446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2056}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2111","reported_seq":2169,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.137940+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:10.137940+0000","last_peered":"2024-09-15T03:06:10.137940+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:10.137940+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17248683,"num_objects":2106,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":16852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":28}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2019","reported_seq":2067,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.116648+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:10.116648+0000","last_peered":"2024-09-15T03:06:10.116648+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:10.116648+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2019,"log_dups_size":0,"ondisk_log_size":2019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16530233,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2018,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":31}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"28'2039","reported_seq":2104,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.709569+0000","last_change":"2024-09-15T03:05:53.237197+0000","last_active":"2024-09-15T03:06:06.709569+0000","last_peered":"2024-09-15T03:06:06.709569+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:05:27.073296+0000","last_became_peered":"2024-09-15T03:05:27.073296+0000","last_unstale":"2024-09-15T03:06:06.709569+0000","last_undegraded":"2024-09-15T03:05:27.065069+0000","last_fullsized":"2024-09-15T03:05:27.064221+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2039,"log_dups_size":0,"ondisk_log_size":2039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16694073,"num_objects":2038,"num_object_clones":0,"num_object_copies":8152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":16303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2059","reported_seq":2109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.072208+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:10.072208+0000","last_peered":"2024-09-15T03:06:10.072208+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:10.072208+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2059,"log_dups_size":0,"ondisk_log_size":2059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16832114,"num_objects":2055,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":19}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"28'1988","reported_seq":2057,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.693068+0000","last_change":"2024-09-15T03:05:54.363436+0000","last_active":"2024-09-15T03:06:06.693068+0000","last_peered":"2024-09-15T03:06:06.693068+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:05:27.074164+0000","last_became_peered":"2024-09-15T03:05:27.074164+0000","last_unstale":"2024-09-15T03:06:06.693068+0000","last_undegraded":"2024-09-15T03:05:27.063361+0000","last_fullsized":"2024-09-15T03:05:27.062449+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1988,"log_dups_size":0,"ondisk_log_size":1988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16258674,"num_objects":1985,"num_object_clones":0,"num_object_copies":7940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1986,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1985}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"28'2040","reported_seq":2090,"reported_epoch":28,"state":"active+undersized+degraded","last_fresh":"2024-09-15T03:06:06.009118+0000","last_change":"2024-09-15T03:05:54.735419+0000","last_active":"2024-09-15T03:06:06.009118+0000","last_peered":"2024-09-15T03:06:06.009118+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:05:27.076127+0000","last_became_peered":"2024-09-15T03:05:27.076127+0000","last_unstale":"2024-09-15T03:06:06.009118+0000","last_undegraded":"2024-09-15T03:05:27.063025+0000","last_fullsized":"2024-09-15T03:05:27.061985+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2040,"log_dups_size":0,"ondisk_log_size":2040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16668274,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2038,"num_write_kb":16292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2035}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":266697276,"num_objects":32560,"num_object_clones":0,"num_object_copies":130240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32395,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32587,"num_write_kb":260578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399347712,"data_stored":399347712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32614,"ondisk_log_size":32614,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":438384,"kb_used_data":132200,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93933456,"statfs":{"total":96636764160,"available":96187858944,"internally_reserved":0,"allocated":135372800,"data_stored":135213822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,4,0,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":431556,"kb_used_data":130040,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93940284,"statfs":{"total":96636764160,"available":96194850816,"internally_reserved":0,"allocated":133160960,"data_stored":133016613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298136,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94073704,"statfs":{"total":96636764160,"available":96331472896,"internally_reserved":0,"allocated":278528,"data_stored":168931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426644,"kb_used_data":124684,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93945196,"statfs":{"total":96636764160,"available":96199880704,"internally_reserved":0,"allocated":127676416,"data_stored":127532827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130748416,"data_stored":130748416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133328896,"data_stored":133328896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135155712,"data_stored":135155712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:13.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 37 20 34314 34294 7.23964 3.22656 0.0182517 0.0215546 2024-09-15T03:06:14.354 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 38 20 34738 34718 7.1363 3.3125 0.0136968 0.021875 2024-09-15T03:06:15.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 39 20 35152 35132 7.03604 3.23438 0.0113464 0.0221884 2024-09-15T03:06:15.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:16.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:06:16.354900+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0224885 2024-09-15T03:06:16.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:06:16.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 40 20 35535 35515 6.93495 2.99219 0.0156879 0.0224885 2024-09-15T03:06:16.387 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:16.407 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-09-15T03:06:14.651097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":272710204,"num_objects":33294,"num_object_clones":0,"num_object_copies":133176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33321,"num_write_kb":266450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33348,"ondisk_log_size":33348,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1617684,"kb_used_data":401840,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375869676,"statfs":{"total":386547056640,"available":384890548224,"internally_reserved":0,"allocated":411484160,"data_stored":410851394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,3,1,0,0,4,0,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12656640,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1545,"num_write_kb":12360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.815253"},"pg_stats":[{"pgid":"1.f","version":"31'2002","reported_seq":2044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.109008+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:10.109008+0000","last_peered":"2024-09-15T03:06:10.109008+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:10.109008+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2002,"log_dups_size":0,"ondisk_log_size":2002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16400384,"num_objects":2002,"num_object_clones":0,"num_object_copies":8008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2002,"num_write_kb":16016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2034","reported_seq":2096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.906371+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:10.906371+0000","last_peered":"2024-09-15T03:06:10.906371+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:10.906371+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2034,"log_dups_size":0,"ondisk_log_size":2034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16627314,"num_objects":2030,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2032,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":32}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2174","reported_seq":2255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:11.671283+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:11.671283+0000","last_peered":"2024-09-15T03:06:11.671283+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:11.671283+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17783609,"num_objects":2171,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":17374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":69}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'1995","reported_seq":2058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.951005+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:10.951005+0000","last_peered":"2024-09-15T03:06:10.951005+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:10.951005+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":1995,"log_dups_size":0,"ondisk_log_size":1995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16307826,"num_objects":1991,"num_object_clones":0,"num_object_copies":7964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":15933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":48}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2136","reported_seq":2192,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:09.992397+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:09.992397+0000","last_peered":"2024-09-15T03:06:09.992397+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:09.992397+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17471090,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2134,"num_write_kb":17069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":26}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2080","reported_seq":2124,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.084924+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:10.084924+0000","last_peered":"2024-09-15T03:06:10.084924+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:10.084924+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2080,"log_dups_size":0,"ondisk_log_size":2080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039360,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2080,"num_write_kb":16640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":24}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2099","reported_seq":2164,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.951049+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:10.951049+0000","last_peered":"2024-09-15T03:06:10.951049+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:10.951049+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17143410,"num_objects":2093,"num_object_clones":0,"num_object_copies":8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2097,"num_write_kb":16756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":41}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2133","reported_seq":2196,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.968145+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:10.968145+0000","last_peered":"2024-09-15T03:06:10.968145+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:10.968145+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2133,"log_dups_size":0,"ondisk_log_size":2133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17430130,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2131,"num_write_kb":17036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2071","reported_seq":2128,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.042921+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:10.042921+0000","last_peered":"2024-09-15T03:06:10.042921+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:10.042921+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2071,"log_dups_size":0,"ondisk_log_size":2071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16895204,"num_objects":2063,"num_object_clones":0,"num_object_copies":8252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2068,"num_write_kb":16521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":21}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2139","reported_seq":2222,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:11.682679+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:11.682679+0000","last_peered":"2024-09-15T03:06:11.682679+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:11.682679+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17503858,"num_objects":2137,"num_object_clones":0,"num_object_copies":8548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2137,"num_write_kb":17094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":79}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2111","reported_seq":2169,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.137940+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:10.137940+0000","last_peered":"2024-09-15T03:06:10.137940+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:10.137940+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17248683,"num_objects":2106,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":16852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":28}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2019","reported_seq":2067,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.116648+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:10.116648+0000","last_peered":"2024-09-15T03:06:10.116648+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:10.116648+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2019,"log_dups_size":0,"ondisk_log_size":2019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16530233,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2018,"num_write_kb":16143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":31}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2100","reported_seq":2177,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:11.713443+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:11.713443+0000","last_peered":"2024-09-15T03:06:11.713443+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:11.713443+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17193785,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":16791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":60}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2059","reported_seq":2109,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.072208+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:10.072208+0000","last_peered":"2024-09-15T03:06:10.072208+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:10.072208+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2059,"log_dups_size":0,"ondisk_log_size":2059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16832114,"num_objects":2055,"num_object_clones":0,"num_object_copies":8220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":16452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":19}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2070","reported_seq":2151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:11.734724+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:11.734724+0000","last_peered":"2024-09-15T03:06:11.734724+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:11.734724+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2070,"log_dups_size":0,"ondisk_log_size":2070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16930418,"num_objects":2067,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2068,"num_write_kb":16541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":79}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2126","reported_seq":2189,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:10.869083+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:10.869083+0000","last_peered":"2024-09-15T03:06:10.869083+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:10.869083+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2126,"log_dups_size":0,"ondisk_log_size":2126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17372786,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2124,"num_write_kb":16980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":39}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":272710204,"num_objects":33294,"num_object_clones":0,"num_object_copies":133176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33321,"num_write_kb":266450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":40960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":412516352,"data_stored":412516352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33348,"ondisk_log_size":33348,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":438384,"kb_used_data":132200,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93933456,"statfs":{"total":96636764160,"available":96187858944,"internally_reserved":0,"allocated":135372800,"data_stored":135213822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,4,0,7],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440316,"kb_used_data":134644,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93931524,"statfs":{"total":96636764160,"available":96185880576,"internally_reserved":0,"allocated":137875456,"data_stored":137719624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300572,"kb_used_data":2704,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94071268,"statfs":{"total":96636764160,"available":96328978432,"internally_reserved":0,"allocated":2768896,"data_stored":2609918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":438412,"kb_used_data":132292,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93933428,"statfs":{"total":96636764160,"available":96187830272,"internally_reserved":0,"allocated":135467008,"data_stored":135308030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.0910000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.657}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":136609792,"data_stored":136609792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2904064,"data_stored":2904064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":137846784,"data_stored":137846784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":135155712,"data_stored":135155712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:16.408 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:17.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 41 20 35934 35914 6.84184 3.11719 0.059575 0.022814 2024-09-15T03:06:17.552 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:17.583 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:17.609 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-09-15T03:06:16.651521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285366844,"num_objects":34839,"num_object_clones":0,"num_object_copies":139356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34866,"num_write_kb":278810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":294912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34893,"ondisk_log_size":34893,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1657644,"kb_used_data":425164,"kb_used_omap":14,"kb_used_meta":1232433,"kb_avail":375829716,"statfs":{"total":386547056640,"available":384849629184,"internally_reserved":0,"allocated":435367936,"data_stored":434735170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1262011419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,11,15,2,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25313280,"num_objects":3090,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3090,"num_write_kb":24720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":294912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.821308"},"pg_stats":[{"pgid":"1.f","version":"31'2144","reported_seq":2186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.106550+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:15.106550+0000","last_peered":"2024-09-15T03:06:15.106550+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:15.106550+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2144,"log_dups_size":0,"ondisk_log_size":2144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":2144,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":17152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2182","reported_seq":2244,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.924263+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:15.924263+0000","last_peered":"2024-09-15T03:06:15.924263+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:15.924263+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839730,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2180,"num_write_kb":17436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2174","reported_seq":2255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:11.671283+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:11.671283+0000","last_peered":"2024-09-15T03:06:11.671283+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:11.671283+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17783609,"num_objects":2171,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":17374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":69}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2120","reported_seq":2183,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.832647+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:15.832647+0000","last_peered":"2024-09-15T03:06:15.832647+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:15.832647+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2120,"log_dups_size":0,"ondisk_log_size":2120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17331826,"num_objects":2116,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2118,"num_write_kb":16933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":173}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2248","reported_seq":2304,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.112603+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:15.112603+0000","last_peered":"2024-09-15T03:06:15.112603+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:15.112603+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2248,"log_dups_size":0,"ondisk_log_size":2248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18388594,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":17965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2201","reported_seq":2245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.174057+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:15.174057+0000","last_peered":"2024-09-15T03:06:15.174057+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:15.174057+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2201,"log_dups_size":0,"ondisk_log_size":2201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18030592,"num_objects":2201,"num_object_clones":0,"num_object_copies":8804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2234","reported_seq":2299,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.925760+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:15.925760+0000","last_peered":"2024-09-15T03:06:15.925760+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:15.925760+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18249330,"num_objects":2228,"num_object_clones":0,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":17836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":176}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2250","reported_seq":2313,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.992291+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:15.992291+0000","last_peered":"2024-09-15T03:06:15.992291+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:15.992291+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2250,"log_dups_size":0,"ondisk_log_size":2250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18388594,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2248,"num_write_kb":17972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2199","reported_seq":2255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.171347+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:15.171347+0000","last_peered":"2024-09-15T03:06:15.171347+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:15.171347+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2199,"log_dups_size":0,"ondisk_log_size":2199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17943780,"num_objects":2191,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2196,"num_write_kb":17545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2139","reported_seq":2222,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:11.682679+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:11.682679+0000","last_peered":"2024-09-15T03:06:11.682679+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:11.682679+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17503858,"num_objects":2137,"num_object_clones":0,"num_object_copies":8548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2137,"num_write_kb":17094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":79}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2253","reported_seq":2311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.128618+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:15.128618+0000","last_peered":"2024-09-15T03:06:15.128618+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:15.128618+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18411947,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2146","reported_seq":2255,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.151269+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:15.151269+0000","last_peered":"2024-09-15T03:06:15.151269+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:15.151269+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17570617,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":17159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":294912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":189}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2100","reported_seq":2177,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:11.713443+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:11.713443+0000","last_peered":"2024-09-15T03:06:11.713443+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:11.713443+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2100,"log_dups_size":0,"ondisk_log_size":2100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17193785,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":16791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":60}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2193","reported_seq":2243,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.085428+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:15.085428+0000","last_peered":"2024-09-15T03:06:15.085428+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:15.085428+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17929842,"num_objects":2189,"num_object_clones":0,"num_object_copies":8756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":17524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2070","reported_seq":2151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:11.734724+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:11.734724+0000","last_peered":"2024-09-15T03:06:11.734724+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:11.734724+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2070,"log_dups_size":0,"ondisk_log_size":2070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16930418,"num_objects":2067,"num_object_clones":0,"num_object_copies":8268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2068,"num_write_kb":16541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":79}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2240","reported_seq":2303,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.963673+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:15.963673+0000","last_peered":"2024-09-15T03:06:15.963673+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:15.963673+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2240,"log_dups_size":0,"ondisk_log_size":2240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18306674,"num_objects":2235,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2238,"num_write_kb":17892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":285366844,"num_objects":34839,"num_object_clones":0,"num_object_copies":139356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34866,"num_write_kb":278810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":294912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":438050816,"data_stored":438050816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34893,"ondisk_log_size":34893,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449304,"kb_used_data":138960,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93922536,"statfs":{"total":96636764160,"available":96176676864,"internally_reserved":0,"allocated":142295040,"data_stored":142136062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,9,4,1,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5250000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440316,"kb_used_data":134644,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93931524,"statfs":{"total":96636764160,"available":96185880576,"internally_reserved":0,"allocated":137875456,"data_stored":137719624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317244,"kb_used_data":11060,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94054596,"statfs":{"total":96636764160,"available":96311906304,"internally_reserved":0,"allocated":11325440,"data_stored":11166462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5010000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8250000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4829999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450780,"kb_used_data":140500,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93921060,"statfs":{"total":96636764160,"available":96175165440,"internally_reserved":0,"allocated":143872000,"data_stored":143713022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145018880,"data_stored":145018880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11497472,"data_stored":11497472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":137846784,"data_stored":137846784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":143687680,"data_stored":143687680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:17.610 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2144", 'reported_seq': 2186, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.106550+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:06:15.106550+0000', 'last_peered': '2024-09-15T03:06:15.106550+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:06:15.106550+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2144, 'log_dups_size': 0, 'ondisk_log_size': 2144, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17563648, 'num_objects': 2144, 'num_object_clones': 0, 'num_object_copies': 8576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2144, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2144, 'num_write_kb': 17152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 168}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2182", 'reported_seq': 2244, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.924263+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:06:15.924263+0000', 'last_peered': '2024-09-15T03:06:15.924263+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:06:15.924263+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2182, 'log_dups_size': 0, 'ondisk_log_size': 2182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17839730, 'num_objects': 2178, 'num_object_clones': 0, 'num_object_copies': 8712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2180, 'num_write_kb': 17436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2174", 'reported_seq': 2255, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:11.671283+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:06:11.671283+0000', 'last_peered': '2024-09-15T03:06:11.671283+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:06:11.671283+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2174, 'log_dups_size': 0, 'ondisk_log_size': 2174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17783609, 'num_objects': 2171, 'num_object_clones': 0, 'num_object_copies': 8684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2173, 'num_write_kb': 17374, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 69}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2120", 'reported_seq': 2183, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.832647+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:06:15.832647+0000', 'last_peered': '2024-09-15T03:06:15.832647+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:06:15.832647+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2120, 'log_dups_size': 0, 'ondisk_log_size': 2120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17331826, 'num_objects': 2116, 'num_object_clones': 0, 'num_object_copies': 8464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2118, 'num_write_kb': 16933, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 173}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:06:17.611 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2248", 'reported_seq': 2304, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.112603+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:06:15.112603+0000', 'last_peered': '2024-09-15T03:06:15.112603+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:06:15.112603+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2248, 'log_dups_size': 0, 'ondisk_log_size': 2248, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18388594, 'num_objects': 2245, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2246, 'num_write_kb': 17965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 138}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2201", 'reported_seq': 2245, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.174057+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:06:15.174057+0000', 'last_peered': '2024-09-15T03:06:15.174057+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:06:15.174057+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2201, 'log_dups_size': 0, 'ondisk_log_size': 2201, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18030592, 'num_objects': 2201, 'num_object_clones': 0, 'num_object_copies': 8804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2201, 'num_write_kb': 17608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 145}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2234", 'reported_seq': 2299, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.925760+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:06:15.925760+0000', 'last_peered': '2024-09-15T03:06:15.925760+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:06:15.925760+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2234, 'log_dups_size': 0, 'ondisk_log_size': 2234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18249330, 'num_objects': 2228, 'num_object_clones': 0, 'num_object_copies': 8912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2228, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2232, 'num_write_kb': 17836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 176}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2250", 'reported_seq': 2313, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.992291+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:06:15.992291+0000', 'last_peered': '2024-09-15T03:06:15.992291+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:06:15.992291+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2250, 'log_dups_size': 0, 'ondisk_log_size': 2250, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18388594, 'num_objects': 2245, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2248, 'num_write_kb': 17972, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2199", 'reported_seq': 2255, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.171347+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:06:15.171347+0000', 'last_peered': '2024-09-15T03:06:15.171347+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:06:15.171347+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2199, 'log_dups_size': 0, 'ondisk_log_size': 2199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17943780, 'num_objects': 2191, 'num_object_clones': 0, 'num_object_copies': 8764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2196, 'num_write_kb': 17545, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 149}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2139", 'reported_seq': 2222, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:11.682679+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:06:11.682679+0000', 'last_peered': '2024-09-15T03:06:11.682679+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:06:11.682679+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2139, 'log_dups_size': 0, 'ondisk_log_size': 2139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17503858, 'num_objects': 2137, 'num_object_clones': 0, 'num_object_copies': 8548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2137, 'num_write_kb': 17094, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 79}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:17.612 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2253", 'reported_seq': 2311, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.128618+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:06:15.128618+0000', 'last_peered': '2024-09-15T03:06:15.128618+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:06:15.128618+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2253, 'log_dups_size': 0, 'ondisk_log_size': 2253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18411947, 'num_objects': 2248, 'num_object_clones': 0, 'num_object_copies': 8992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2250, 'num_write_kb': 17988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 170}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2146", 'reported_seq': 2255, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.151269+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:06:15.151269+0000', 'last_peered': '2024-09-15T03:06:15.151269+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:06:15.151269+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2146, 'log_dups_size': 0, 'ondisk_log_size': 2146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17570617, 'num_objects': 2145, 'num_object_clones': 0, 'num_object_copies': 8580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1956, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2145, 'num_write_kb': 17159, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 294912, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 189}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2100", 'reported_seq': 2177, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:11.713443+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:06:11.713443+0000', 'last_peered': '2024-09-15T03:06:11.713443+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:06:11.713443+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2100, 'log_dups_size': 0, 'ondisk_log_size': 2100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17193785, 'num_objects': 2099, 'num_object_clones': 0, 'num_object_copies': 8396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2099, 'num_write_kb': 16791, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 60}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2193", 'reported_seq': 2243, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.085428+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:06:15.085428+0000', 'last_peered': '2024-09-15T03:06:15.085428+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:06:15.085428+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2193, 'log_dups_size': 0, 'ondisk_log_size': 2193, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17929842, 'num_objects': 2189, 'num_object_clones': 0, 'num_object_copies': 8756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2191, 'num_write_kb': 17524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 153}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2070", 'reported_seq': 2151, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:11.734724+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:06:11.734724+0000', 'last_peered': '2024-09-15T03:06:11.734724+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:06:11.734724+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2070, 'log_dups_size': 0, 'ondisk_log_size': 2070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16930418, 'num_objects': 2067, 'num_object_clones': 0, 'num_object_copies': 8268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2068, 'num_write_kb': 16541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 79}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:06:17.613 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2240", 'reported_seq': 2303, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:15.963673+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:06:15.963673+0000', 'last_peered': '2024-09-15T03:06:15.963673+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:06:15.963673+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2240, 'log_dups_size': 0, 'ondisk_log_size': 2240, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18306674, 'num_objects': 2235, 'num_object_clones': 0, 'num_object_copies': 8940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2238, 'num_write_kb': 17892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 153}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:17.614 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:06:18.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 42 20 36326 36306 6.75186 3.0625 0.0293204 0.0231211 2024-09-15T03:06:18.949 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:18.949 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32620/141512 objects degraded (23.051%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":102,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35378,"data_bytes":289782332,"bytes_used":1697411072,"bytes_avail":384849645568,"bytes_total":386547056640,"degraded_objects":32620,"degraded_total":141512,"degraded_ratio":0.2305104867431737,"recovering_objects_per_sec":3,"recovering_bytes_per_sec":29479,"recovering_keys_per_sec":0,"num_objects_recovered":36,"num_bytes_recovered":294912,"num_keys_recovered":0,"write_bytes_sec":2971715,"read_op_per_sec":0,"write_op_per_sec":362},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-15T03:06:16.651627+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T03:06:18.985 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:06:18.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:19.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 43 20 36721 36701 6.6666 3.08594 0.0270358 0.0234161 2024-09-15T03:06:20.356 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 44 20 37107 37087 6.58362 3.01562 0.0328021 0.023707 2024-09-15T03:06:20.445 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:20.459 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:20.474 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-15T03:06:18.652008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289782332,"num_objects":35378,"num_object_clones":0,"num_object_copies":141512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35405,"num_write_kb":283122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":294912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35432,"ondisk_log_size":35432,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1657628,"kb_used_data":433468,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375829732,"statfs":{"total":386547056640,"available":384849645568,"internally_reserved":0,"allocated":443871232,"data_stored":443238466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[12,0,12,17,4,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29728768,"num_objects":3629,"num_object_clones":0,"num_object_copies":14516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3629,"num_write_kb":29032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":294912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003908"},"pg_stats":[{"pgid":"1.f","version":"31'2144","reported_seq":2186,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.106550+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:15.106550+0000","last_peered":"2024-09-15T03:06:15.106550+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:15.106550+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2144,"log_dups_size":0,"ondisk_log_size":2144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":2144,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2144,"num_write_kb":17152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2182","reported_seq":2244,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.924263+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:15.924263+0000","last_peered":"2024-09-15T03:06:15.924263+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:15.924263+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17839730,"num_objects":2178,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2180,"num_write_kb":17436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2310","reported_seq":2391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:16.725023+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:16.725023+0000","last_peered":"2024-09-15T03:06:16.725023+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:16.725023+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18897721,"num_objects":2307,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2309,"num_write_kb":18462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2120","reported_seq":2183,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.832647+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:15.832647+0000","last_peered":"2024-09-15T03:06:15.832647+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:15.832647+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2120,"log_dups_size":0,"ondisk_log_size":2120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17331826,"num_objects":2116,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2118,"num_write_kb":16933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":173}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2248","reported_seq":2304,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.112603+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:15.112603+0000","last_peered":"2024-09-15T03:06:15.112603+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:15.112603+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2248,"log_dups_size":0,"ondisk_log_size":2248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18388594,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2246,"num_write_kb":17965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2201","reported_seq":2245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.174057+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:15.174057+0000","last_peered":"2024-09-15T03:06:15.174057+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:15.174057+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2201,"log_dups_size":0,"ondisk_log_size":2201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18030592,"num_objects":2201,"num_object_clones":0,"num_object_copies":8804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":17608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2234","reported_seq":2299,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.925760+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:15.925760+0000","last_peered":"2024-09-15T03:06:15.925760+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:15.925760+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18249330,"num_objects":2228,"num_object_clones":0,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":17836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":176}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2250","reported_seq":2313,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.992291+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:15.992291+0000","last_peered":"2024-09-15T03:06:15.992291+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:15.992291+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2250,"log_dups_size":0,"ondisk_log_size":2250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18388594,"num_objects":2245,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2248,"num_write_kb":17972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2199","reported_seq":2255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.171347+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:15.171347+0000","last_peered":"2024-09-15T03:06:15.171347+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:15.171347+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2199,"log_dups_size":0,"ondisk_log_size":2199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17943780,"num_objects":2191,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2196,"num_write_kb":17545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2260","reported_seq":2342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:16.764593+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:16.764593+0000","last_peered":"2024-09-15T03:06:16.764593+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:16.764593+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2260,"log_dups_size":0,"ondisk_log_size":2260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18495090,"num_objects":2258,"num_object_clones":0,"num_object_copies":9032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":18062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2253","reported_seq":2311,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.128618+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:15.128618+0000","last_peered":"2024-09-15T03:06:15.128618+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:15.128618+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2253,"log_dups_size":0,"ondisk_log_size":2253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18411947,"num_objects":2248,"num_object_clones":0,"num_object_copies":8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":17988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2146","reported_seq":2255,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.151269+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:15.151269+0000","last_peered":"2024-09-15T03:06:15.151269+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:15.151269+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17570617,"num_objects":2145,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":17159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":294912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":189}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2242","reported_seq":2319,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:16.757875+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:16.757875+0000","last_peered":"2024-09-15T03:06:16.757875+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:16.757875+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2242,"log_dups_size":0,"ondisk_log_size":2242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18357049,"num_objects":2241,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":17927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":202}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2193","reported_seq":2243,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.085428+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:15.085428+0000","last_peered":"2024-09-15T03:06:15.085428+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:15.085428+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17929842,"num_objects":2189,"num_object_clones":0,"num_object_copies":8756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":17524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2210","reported_seq":2289,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:16.788654+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:16.788654+0000","last_peered":"2024-09-15T03:06:16.788654+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:16.788654+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18077298,"num_objects":2207,"num_object_clones":0,"num_object_copies":8828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":17661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":219}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2240","reported_seq":2303,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:15.963673+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:15.963673+0000","last_peered":"2024-09-15T03:06:15.963673+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:15.963673+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2240,"log_dups_size":0,"ondisk_log_size":2240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18306674,"num_objects":2235,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2238,"num_write_kb":17892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":153}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":289782332,"num_objects":35378,"num_object_clones":0,"num_object_copies":141512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35405,"num_write_kb":283122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":294912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":446455808,"data_stored":446455808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35432,"ondisk_log_size":35432,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":449304,"kb_used_data":138960,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93922536,"statfs":{"total":96636764160,"available":96176676864,"internally_reserved":0,"allocated":142295040,"data_stored":142136062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,9,4,1,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5250000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":440300,"kb_used_data":142948,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93931540,"statfs":{"total":96636764160,"available":96185896960,"internally_reserved":0,"allocated":146378752,"data_stored":146222920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.3170000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317244,"kb_used_data":11060,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94054596,"statfs":{"total":96636764160,"available":96311906304,"internally_reserved":0,"allocated":11325440,"data_stored":11166462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5010000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8250000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.4829999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450780,"kb_used_data":140500,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93921060,"statfs":{"total":96636764160,"available":96175165440,"internally_reserved":0,"allocated":143872000,"data_stored":143713022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145018880,"data_stored":145018880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11497472,"data_stored":11497472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":146251776,"data_stored":146251776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":143687680,"data_stored":143687680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:21.357 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 45 20 37509 37489 6.50694 3.14062 0.0324064 0.0239864 2024-09-15T03:06:22.358 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 46 20 37908 37888 6.43325 3.11719 0.0383771 0.0242618 2024-09-15T03:06:23.358 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 47 20 38322 38302 6.36519 3.23438 0.0159963 0.0245188 2024-09-15T03:06:23.477 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:24.358 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 48 20 38703 38683 6.2946 2.97656 0.0653549 0.0248068 2024-09-15T03:06:24.558 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:24.570 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:24.583 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-15T03:06:22.652966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305789500,"num_objects":37332,"num_object_clones":0,"num_object_copies":149328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37359,"num_write_kb":298754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37386,"ondisk_log_size":37386,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1682452,"kb_used_data":466612,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375804908,"statfs":{"total":386547056640,"available":384824225792,"internally_reserved":0,"allocated":477810688,"data_stored":477177922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,7,14,12,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39092224,"num_objects":4772,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":38176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004183"},"pg_stats":[{"pgid":"1.f","version":"31'2256","reported_seq":2295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.096061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:20.096061+0000","last_peered":"2024-09-15T03:06:20.096061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:20.096061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481152,"num_objects":2256,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":18048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2323","reported_seq":2385,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.900337+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:20.900337+0000","last_peered":"2024-09-15T03:06:20.900337+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:20.900337+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2323,"log_dups_size":0,"ondisk_log_size":2323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18994802,"num_objects":2319,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2321,"num_write_kb":18564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2443","reported_seq":2521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.817367+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:21.817367+0000","last_peered":"2024-09-15T03:06:21.817367+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:21.817367+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2443,"log_dups_size":0,"ondisk_log_size":2443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19987257,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":19526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2230","reported_seq":2293,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.984899+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:20.984899+0000","last_peered":"2024-09-15T03:06:20.984899+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:20.984899+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2230,"log_dups_size":0,"ondisk_log_size":2230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18232946,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":17813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2369","reported_seq":2425,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.185969+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:20.185969+0000","last_peered":"2024-09-15T03:06:20.185969+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:20.185969+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2369,"log_dups_size":0,"ondisk_log_size":2369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379826,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2367,"num_write_kb":18933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2324","reported_seq":2367,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.162895+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:20.162895+0000","last_peered":"2024-09-15T03:06:20.162895+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:20.162895+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2324,"log_dups_size":0,"ondisk_log_size":2324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19038208,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2358","reported_seq":2423,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.975848+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:20.975848+0000","last_peered":"2024-09-15T03:06:20.975848+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:20.975848+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2358,"log_dups_size":0,"ondisk_log_size":2358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19265138,"num_objects":2352,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2356,"num_write_kb":18828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2375","reported_seq":2438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.986319+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:20.986319+0000","last_peered":"2024-09-15T03:06:20.986319+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:20.986319+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2375,"log_dups_size":0,"ondisk_log_size":2375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19412594,"num_objects":2370,"num_object_clones":0,"num_object_copies":9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2373,"num_write_kb":18972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2331","reported_seq":2388,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.095896+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:20.095896+0000","last_peered":"2024-09-15T03:06:20.095896+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:20.095896+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2331,"log_dups_size":0,"ondisk_log_size":2331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19025124,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2328,"num_write_kb":18601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2389","reported_seq":2472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.737931+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:21.737931+0000","last_peered":"2024-09-15T03:06:21.737931+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:21.737931+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2389,"log_dups_size":0,"ondisk_log_size":2389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19551858,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":19094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":329}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2382","reported_seq":2439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.163075+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:20.163075+0000","last_peered":"2024-09-15T03:06:20.163075+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:20.163075+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2382,"log_dups_size":0,"ondisk_log_size":2382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19468715,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":19020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2274","reported_seq":2462,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.133675+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:20.133675+0000","last_peered":"2024-09-15T03:06:20.133675+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:20.133675+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2274,"log_dups_size":0,"ondisk_log_size":2274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18619193,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2273,"num_write_kb":18183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":356}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2352","reported_seq":2429,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.769940+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:21.769940+0000","last_peered":"2024-09-15T03:06:21.769940+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:21.769940+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2352,"log_dups_size":0,"ondisk_log_size":2352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19258169,"num_objects":2351,"num_object_clones":0,"num_object_copies":9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2351,"num_write_kb":18807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2300","reported_seq":2350,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.133095+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:20.133095+0000","last_peered":"2024-09-15T03:06:20.133095+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:20.133095+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2300,"log_dups_size":0,"ondisk_log_size":2300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18806386,"num_objects":2296,"num_object_clones":0,"num_object_copies":9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2298,"num_write_kb":18380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":260}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2326","reported_seq":2407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.760671+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:21.760671+0000","last_peered":"2024-09-15T03:06:21.760671+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:21.760671+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2326,"log_dups_size":0,"ondisk_log_size":2326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19027570,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2354","reported_seq":2417,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.022997+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:21.022997+0000","last_peered":"2024-09-15T03:06:21.022997+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:21.022997+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19240562,"num_objects":2349,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":18804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":305789500,"num_objects":37332,"num_object_clones":0,"num_object_copies":149328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37359,"num_write_kb":298754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":478982144,"data_stored":478982144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37386,"ondisk_log_size":37386,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450448,"kb_used_data":148424,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93921392,"statfs":{"total":96636764160,"available":96175505408,"internally_reserved":0,"allocated":151986176,"data_stored":151827198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,6,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.923}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":452436,"kb_used_data":150924,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93919404,"statfs":{"total":96636764160,"available":96173469696,"internally_reserved":0,"allocated":154546176,"data_stored":154390344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329176,"kb_used_data":18832,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94042664,"statfs":{"total":96636764160,"available":96299687936,"internally_reserved":0,"allocated":19283968,"data_stored":19124990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1620000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450392,"kb_used_data":148432,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93921448,"statfs":{"total":96636764160,"available":96175562752,"internally_reserved":0,"allocated":151994368,"data_stored":151835390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.048}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":6.8659999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153100288,"data_stored":153100288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19775488,"data_stored":19775488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154353664,"data_stored":154353664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151752704,"data_stored":151752704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:24.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:25.358 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 49 20 39076 39056 6.22562 2.91406 0.117645 0.0250693 2024-09-15T03:06:25.738 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:25.765 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:25.779 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-15T03:06:24.653424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305789500,"num_objects":37332,"num_object_clones":0,"num_object_copies":149328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37359,"num_write_kb":298754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37386,"ondisk_log_size":37386,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1682452,"kb_used_data":466612,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375804908,"statfs":{"total":386547056640,"available":384824225792,"internally_reserved":0,"allocated":477810688,"data_stored":477177922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,7,14,12,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-70,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4038,"num_write_kb":32304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002853"},"pg_stats":[{"pgid":"1.f","version":"31'2256","reported_seq":2295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.096061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:20.096061+0000","last_peered":"2024-09-15T03:06:20.096061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:20.096061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2256,"log_dups_size":0,"ondisk_log_size":2256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18481152,"num_objects":2256,"num_object_clones":0,"num_object_copies":9024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2256,"num_write_kb":18048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":280}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2323","reported_seq":2385,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.900337+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:20.900337+0000","last_peered":"2024-09-15T03:06:20.900337+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:20.900337+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2323,"log_dups_size":0,"ondisk_log_size":2323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18994802,"num_objects":2319,"num_object_clones":0,"num_object_copies":9276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2321,"num_write_kb":18564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2443","reported_seq":2521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.817367+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:21.817367+0000","last_peered":"2024-09-15T03:06:21.817367+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:21.817367+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2443,"log_dups_size":0,"ondisk_log_size":2443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19987257,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":19526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2230","reported_seq":2293,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.984899+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:20.984899+0000","last_peered":"2024-09-15T03:06:20.984899+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:20.984899+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2230,"log_dups_size":0,"ondisk_log_size":2230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18232946,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":17813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2369","reported_seq":2425,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.185969+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:20.185969+0000","last_peered":"2024-09-15T03:06:20.185969+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:20.185969+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2369,"log_dups_size":0,"ondisk_log_size":2369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379826,"num_objects":2366,"num_object_clones":0,"num_object_copies":9464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2367,"num_write_kb":18933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2324","reported_seq":2367,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.162895+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:20.162895+0000","last_peered":"2024-09-15T03:06:20.162895+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:20.162895+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2324,"log_dups_size":0,"ondisk_log_size":2324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19038208,"num_objects":2324,"num_object_clones":0,"num_object_copies":9296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2358","reported_seq":2423,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.975848+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:20.975848+0000","last_peered":"2024-09-15T03:06:20.975848+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:20.975848+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2358,"log_dups_size":0,"ondisk_log_size":2358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19265138,"num_objects":2352,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2356,"num_write_kb":18828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":300}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2375","reported_seq":2438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.986319+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:20.986319+0000","last_peered":"2024-09-15T03:06:20.986319+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:20.986319+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2375,"log_dups_size":0,"ondisk_log_size":2375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19412594,"num_objects":2370,"num_object_clones":0,"num_object_copies":9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2373,"num_write_kb":18972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2331","reported_seq":2388,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.095896+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:20.095896+0000","last_peered":"2024-09-15T03:06:20.095896+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:20.095896+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2331,"log_dups_size":0,"ondisk_log_size":2331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19025124,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2328,"num_write_kb":18601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":281}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2389","reported_seq":2472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.737931+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:21.737931+0000","last_peered":"2024-09-15T03:06:21.737931+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:21.737931+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2389,"log_dups_size":0,"ondisk_log_size":2389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19551858,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":19094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":329}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2382","reported_seq":2439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.163075+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:20.163075+0000","last_peered":"2024-09-15T03:06:20.163075+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:20.163075+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2382,"log_dups_size":0,"ondisk_log_size":2382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19468715,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":19020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2274","reported_seq":2462,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.133675+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:20.133675+0000","last_peered":"2024-09-15T03:06:20.133675+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:20.133675+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2274,"log_dups_size":0,"ondisk_log_size":2274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18619193,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1917,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2273,"num_write_kb":18183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":356}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2352","reported_seq":2429,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.769940+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:21.769940+0000","last_peered":"2024-09-15T03:06:21.769940+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:21.769940+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2352,"log_dups_size":0,"ondisk_log_size":2352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19258169,"num_objects":2351,"num_object_clones":0,"num_object_copies":9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2351,"num_write_kb":18807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2300","reported_seq":2350,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:20.133095+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:20.133095+0000","last_peered":"2024-09-15T03:06:20.133095+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:20.133095+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2300,"log_dups_size":0,"ondisk_log_size":2300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18806386,"num_objects":2296,"num_object_clones":0,"num_object_copies":9184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2298,"num_write_kb":18380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":260}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2326","reported_seq":2407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.760671+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:21.760671+0000","last_peered":"2024-09-15T03:06:21.760671+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:21.760671+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2326,"log_dups_size":0,"ondisk_log_size":2326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19027570,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2354","reported_seq":2417,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.022997+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:21.022997+0000","last_peered":"2024-09-15T03:06:21.022997+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:21.022997+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2354,"log_dups_size":0,"ondisk_log_size":2354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19240562,"num_objects":2349,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":18804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":305789500,"num_objects":37332,"num_object_clones":0,"num_object_copies":149328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37359,"num_write_kb":298754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":614400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":478982144,"data_stored":478982144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37386,"ondisk_log_size":37386,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450448,"kb_used_data":148424,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93921392,"statfs":{"total":96636764160,"available":96175505408,"internally_reserved":0,"allocated":151986176,"data_stored":151827198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,6,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.923}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":452436,"kb_used_data":150924,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93919404,"statfs":{"total":96636764160,"available":96173469696,"internally_reserved":0,"allocated":154546176,"data_stored":154390344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329176,"kb_used_data":18832,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":94042664,"statfs":{"total":96636764160,"available":96299687936,"internally_reserved":0,"allocated":19283968,"data_stored":19124990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.1620000000000008}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":450392,"kb_used_data":148432,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93921448,"statfs":{"total":96636764160,"available":96175562752,"internally_reserved":0,"allocated":151994368,"data_stored":151835390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.048}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":6.8659999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153100288,"data_stored":153100288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19775488,"data_stored":19775488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154353664,"data_stored":154353664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151752704,"data_stored":151752704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:25.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:06:25.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2256", 'reported_seq': 2295, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.096061+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:06:20.096061+0000', 'last_peered': '2024-09-15T03:06:20.096061+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:06:20.096061+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2256, 'log_dups_size': 0, 'ondisk_log_size': 2256, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18481152, 'num_objects': 2256, 'num_object_clones': 0, 'num_object_copies': 9024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2256, 'num_write_kb': 18048, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 280}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:25.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:06:25.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2323", 'reported_seq': 2385, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.900337+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:06:20.900337+0000', 'last_peered': '2024-09-15T03:06:20.900337+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:06:20.900337+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2323, 'log_dups_size': 0, 'ondisk_log_size': 2323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18994802, 'num_objects': 2319, 'num_object_clones': 0, 'num_object_copies': 9276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2321, 'num_write_kb': 18564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 321}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:25.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:06:25.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2443", 'reported_seq': 2521, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:21.817367+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:06:21.817367+0000', 'last_peered': '2024-09-15T03:06:21.817367+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:06:21.817367+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2443, 'log_dups_size': 0, 'ondisk_log_size': 2443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19987257, 'num_objects': 2440, 'num_object_clones': 0, 'num_object_copies': 9760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2442, 'num_write_kb': 19526, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 338}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:25.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:06:25.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2230", 'reported_seq': 2293, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.984899+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:06:20.984899+0000', 'last_peered': '2024-09-15T03:06:20.984899+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:06:20.984899+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2230, 'log_dups_size': 0, 'ondisk_log_size': 2230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18232946, 'num_objects': 2226, 'num_object_clones': 0, 'num_object_copies': 8904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2228, 'num_write_kb': 17813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2369", 'reported_seq': 2425, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.185969+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:06:20.185969+0000', 'last_peered': '2024-09-15T03:06:20.185969+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:06:20.185969+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2369, 'log_dups_size': 0, 'ondisk_log_size': 2369, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19379826, 'num_objects': 2366, 'num_object_clones': 0, 'num_object_copies': 9464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2367, 'num_write_kb': 18933, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 259}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2324", 'reported_seq': 2367, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.162895+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:06:20.162895+0000', 'last_peered': '2024-09-15T03:06:20.162895+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:06:20.162895+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2324, 'log_dups_size': 0, 'ondisk_log_size': 2324, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19038208, 'num_objects': 2324, 'num_object_clones': 0, 'num_object_copies': 9296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2324, 'num_write_kb': 18592, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 268}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2358", 'reported_seq': 2423, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.975848+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:06:20.975848+0000', 'last_peered': '2024-09-15T03:06:20.975848+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:06:20.975848+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2358, 'log_dups_size': 0, 'ondisk_log_size': 2358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19265138, 'num_objects': 2352, 'num_object_clones': 0, 'num_object_copies': 9408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2356, 'num_write_kb': 18828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 300}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2375", 'reported_seq': 2438, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.986319+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:06:20.986319+0000', 'last_peered': '2024-09-15T03:06:20.986319+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:06:20.986319+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2375, 'log_dups_size': 0, 'ondisk_log_size': 2375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19412594, 'num_objects': 2370, 'num_object_clones': 0, 'num_object_copies': 9480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2373, 'num_write_kb': 18972, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:06:25.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2331", 'reported_seq': 2388, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.095896+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:06:20.095896+0000', 'last_peered': '2024-09-15T03:06:20.095896+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:06:20.095896+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2331, 'log_dups_size': 0, 'ondisk_log_size': 2331, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19025124, 'num_objects': 2323, 'num_object_clones': 0, 'num_object_copies': 9292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2328, 'num_write_kb': 18601, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 281}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2389", 'reported_seq': 2472, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:21.737931+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:06:21.737931+0000', 'last_peered': '2024-09-15T03:06:21.737931+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:06:21.737931+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2389, 'log_dups_size': 0, 'ondisk_log_size': 2389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19551858, 'num_objects': 2387, 'num_object_clones': 0, 'num_object_copies': 9548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2387, 'num_write_kb': 19094, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 329}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2382", 'reported_seq': 2439, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.163075+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:06:20.163075+0000', 'last_peered': '2024-09-15T03:06:20.163075+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:06:20.163075+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2382, 'log_dups_size': 0, 'ondisk_log_size': 2382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19468715, 'num_objects': 2377, 'num_object_clones': 0, 'num_object_copies': 9508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2379, 'num_write_kb': 19020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 299}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2274", 'reported_seq': 2462, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.133675+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:06:20.133675+0000', 'last_peered': '2024-09-15T03:06:20.133675+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:06:20.133675+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2274, 'log_dups_size': 0, 'ondisk_log_size': 2274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18619193, 'num_objects': 2273, 'num_object_clones': 0, 'num_object_copies': 9092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1917, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2273, 'num_write_kb': 18183, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 614400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 356}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2352", 'reported_seq': 2429, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:21.769940+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:06:21.769940+0000', 'last_peered': '2024-09-15T03:06:21.769940+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:06:21.769940+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2352, 'log_dups_size': 0, 'ondisk_log_size': 2352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19258169, 'num_objects': 2351, 'num_object_clones': 0, 'num_object_copies': 9404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2351, 'num_write_kb': 18807, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 312}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2300", 'reported_seq': 2350, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:20.133095+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:06:20.133095+0000', 'last_peered': '2024-09-15T03:06:20.133095+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:06:20.133095+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2300, 'log_dups_size': 0, 'ondisk_log_size': 2300, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18806386, 'num_objects': 2296, 'num_object_clones': 0, 'num_object_copies': 9184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2298, 'num_write_kb': 18380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 260}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:25.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:06:25.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2326", 'reported_seq': 2407, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:21.760671+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:06:21.760671+0000', 'last_peered': '2024-09-15T03:06:21.760671+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:06:21.760671+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2326, 'log_dups_size': 0, 'ondisk_log_size': 2326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19027570, 'num_objects': 2323, 'num_object_clones': 0, 'num_object_copies': 9292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2324, 'num_write_kb': 18589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:25.783 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:06:25.783 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2354", 'reported_seq': 2417, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:21.022997+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:06:21.022997+0000', 'last_peered': '2024-09-15T03:06:21.022997+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:06:21.022997+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2354, 'log_dups_size': 0, 'ondisk_log_size': 2354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19240562, 'num_objects': 2349, 'num_object_clones': 0, 'num_object_copies': 9396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2352, 'num_write_kb': 18804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 267}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:25.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:06:26.359 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 50 20 39467 39447 6.16205 3.05469 0.051587 0.0253439 2024-09-15T03:06:27.201 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:27.201 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32529/155232 objects degraded (20.955%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":110,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":38808,"data_bytes":317880892,"bytes_used":1747398656,"bytes_avail":384799657984,"bytes_total":386547056640,"degraded_objects":32529,"degraded_total":155232,"degraded_ratio":0.20955086580086579,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":83266,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"write_bytes_sec":3763368,"read_op_per_sec":0,"write_op_per_sec":459},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T03:06:26.653935+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T03:06:27.255 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:06:27.255 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:27.359 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 51 20 39826 39806 6.09623 2.80469 0.0468937 0.0256072 2024-09-15T03:06:28.359 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 52 20 40195 40175 6.03443 2.88281 0.0138246 0.0258677 2024-09-15T03:06:28.431 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:28.473 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:28.490 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-15T03:06:26.653824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":317880892,"num_objects":38808,"num_object_clones":0,"num_object_copies":155232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38835,"num_write_kb":310562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38862,"ondisk_log_size":38862,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1706444,"kb_used_data":490604,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375780916,"statfs":{"total":386547056640,"available":384799657984,"internally_reserved":0,"allocated":502378496,"data_stored":501745730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,7,6,9,7,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":45170688,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5514,"num_write_kb":44112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"1.f","version":"31'2352","reported_seq":2394,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.161658+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:25.161658+0000","last_peered":"2024-09-15T03:06:25.161658+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:25.161658+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2352,"log_dups_size":0,"ondisk_log_size":2352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19267584,"num_objects":2352,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":18816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2441","reported_seq":2503,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.987713+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:25.987713+0000","last_peered":"2024-09-15T03:06:25.987713+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:25.987713+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2441,"log_dups_size":0,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19961458,"num_objects":2437,"num_object_clones":0,"num_object_copies":9748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2443","reported_seq":2521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.817367+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:21.817367+0000","last_peered":"2024-09-15T03:06:21.817367+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:21.817367+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2443,"log_dups_size":0,"ondisk_log_size":2443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19987257,"num_objects":2440,"num_object_clones":0,"num_object_copies":9760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":19526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2350","reported_seq":2413,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.919098+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:25.919098+0000","last_peered":"2024-09-15T03:06:25.919098+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:25.919098+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2350,"log_dups_size":0,"ondisk_log_size":2350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19215986,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2348,"num_write_kb":18773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2478","reported_seq":2534,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.245303+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:25.245303+0000","last_peered":"2024-09-15T03:06:25.245303+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:25.245303+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2478,"log_dups_size":0,"ondisk_log_size":2478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20272754,"num_objects":2475,"num_object_clones":0,"num_object_copies":9900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":19805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":368}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2465","reported_seq":2509,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.211463+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:25.211463+0000","last_peered":"2024-09-15T03:06:25.211463+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:25.211463+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2465,"log_dups_size":0,"ondisk_log_size":2465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20193280,"num_objects":2465,"num_object_clones":0,"num_object_copies":9860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2465,"num_write_kb":19720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2483","reported_seq":2548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.988676+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:25.988676+0000","last_peered":"2024-09-15T03:06:25.988676+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:25.988676+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2483,"log_dups_size":0,"ondisk_log_size":2483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20289138,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2481,"num_write_kb":19828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2507","reported_seq":2570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.953767+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:25.953767+0000","last_peered":"2024-09-15T03:06:25.953767+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:25.953767+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493938,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":20028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2469","reported_seq":2526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.280622+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:25.280622+0000","last_peered":"2024-09-15T03:06:25.280622+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:25.280622+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2469,"log_dups_size":0,"ondisk_log_size":2469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20155620,"num_objects":2461,"num_object_clones":0,"num_object_copies":9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2466,"num_write_kb":19705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":419}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2389","reported_seq":2472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.737931+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:21.737931+0000","last_peered":"2024-09-15T03:06:21.737931+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:21.737931+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2389,"log_dups_size":0,"ondisk_log_size":2389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19551858,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":19094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":329}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2506","reported_seq":2564,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.241561+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:25.241561+0000","last_peered":"2024-09-15T03:06:25.241561+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:25.241561+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20484523,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":20012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":423}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2392","reported_seq":2684,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.202790+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:25.202790+0000","last_peered":"2024-09-15T03:06:25.202790+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:25.202790+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2392,"log_dups_size":0,"ondisk_log_size":2392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19585849,"num_objects":2391,"num_object_clones":0,"num_object_copies":9564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2391,"num_write_kb":19127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":526}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2352","reported_seq":2429,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.769940+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:21.769940+0000","last_peered":"2024-09-15T03:06:21.769940+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:21.769940+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2352,"log_dups_size":0,"ondisk_log_size":2352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19258169,"num_objects":2351,"num_object_clones":0,"num_object_copies":9404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2351,"num_write_kb":18807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2432","reported_seq":2482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.178806+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:25.178806+0000","last_peered":"2024-09-15T03:06:25.178806+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:25.178806+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2432,"log_dups_size":0,"ondisk_log_size":2432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19887730,"num_objects":2428,"num_object_clones":0,"num_object_copies":9712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":19436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":392}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2326","reported_seq":2407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:21.760671+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:21.760671+0000","last_peered":"2024-09-15T03:06:21.760671+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:21.760671+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2326,"log_dups_size":0,"ondisk_log_size":2326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19027570,"num_objects":2323,"num_object_clones":0,"num_object_copies":9292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2324,"num_write_kb":18589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":335}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2477","reported_seq":2540,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.957982+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:25.957982+0000","last_peered":"2024-09-15T03:06:25.957982+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:25.957982+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20248178,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":19788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":317880892,"num_objects":38808,"num_object_clones":0,"num_object_copies":155232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32529,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38835,"num_write_kb":310562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503455744,"data_stored":503455744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38862,"ondisk_log_size":38862,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462552,"kb_used_data":156368,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93909288,"statfs":{"total":96636764160,"available":96163110912,"internally_reserved":0,"allocated":160120832,"data_stored":159961854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":452436,"kb_used_data":150924,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93919404,"statfs":{"total":96636764160,"available":96173469696,"internally_reserved":0,"allocated":154546176,"data_stored":154390344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.317}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329008,"kb_used_data":26984,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94042832,"statfs":{"total":96636764160,"available":96299859968,"internally_reserved":0,"allocated":27631616,"data_stored":27472638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.091000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462448,"kb_used_data":156328,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93909392,"statfs":{"total":96636764160,"available":96163217408,"internally_reserved":0,"allocated":160079872,"data_stored":159920894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161132544,"data_stored":161132544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28024832,"data_stored":28024832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154353664,"data_stored":154353664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159944704,"data_stored":159944704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:29.360 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 53 20 40589 40569 5.97866 3.07812 0.0149729 0.0261149 2024-09-15T03:06:30.359 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 54 20 40965 40945 5.92234 2.9375 0.0410906 0.0263454 2024-09-15T03:06:31.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 55 20 41361 41341 5.8708 3.09375 0.0875318 0.0265939 2024-09-15T03:06:31.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:32.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 56 20 41737 41717 5.81843 2.9375 0.124489 0.0268316 2024-09-15T03:06:32.571 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:32.688 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:32.689 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-15T03:06:30.656542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328981052,"num_objects":40163,"num_object_clones":0,"num_object_copies":160652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40190,"num_write_kb":321402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40217,"ondisk_log_size":40217,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1731528,"kb_used_data":503208,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375755832,"statfs":{"total":386547056640,"available":384773971968,"internally_reserved":0,"allocated":515284992,"data_stored":514652226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,6,5,8,8,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39198720,"num_objects":4785,"num_object_clones":0,"num_object_copies":19140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4785,"num_write_kb":38280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004534"},"pg_stats":[{"pgid":"1.f","version":"31'2479","reported_seq":2520,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.218787+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:30.218787+0000","last_peered":"2024-09-15T03:06:30.218787+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:30.218787+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2479,"log_dups_size":0,"ondisk_log_size":2479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20307968,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2479,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2441","reported_seq":2503,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.987713+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:25.987713+0000","last_peered":"2024-09-15T03:06:25.987713+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:25.987713+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2441,"log_dups_size":0,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19961458,"num_objects":2437,"num_object_clones":0,"num_object_copies":9748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2566","reported_seq":2647,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:26.705168+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:26.705168+0000","last_peered":"2024-09-15T03:06:26.705168+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:26.705168+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2566,"log_dups_size":0,"ondisk_log_size":2566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20994873,"num_objects":2563,"num_object_clones":0,"num_object_copies":10252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2565,"num_write_kb":20510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2350","reported_seq":2413,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.919098+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:25.919098+0000","last_peered":"2024-09-15T03:06:25.919098+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:25.919098+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2350,"log_dups_size":0,"ondisk_log_size":2350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19215986,"num_objects":2346,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2348,"num_write_kb":18773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2612","reported_seq":2668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.237490+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:30.237490+0000","last_peered":"2024-09-15T03:06:30.237490+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:30.237490+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2612,"log_dups_size":0,"ondisk_log_size":2612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21370482,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2610,"num_write_kb":20877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2604","reported_seq":2648,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.223041+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:30.223041+0000","last_peered":"2024-09-15T03:06:30.223041+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:30.223041+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2604,"log_dups_size":0,"ondisk_log_size":2604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21331968,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2483","reported_seq":2548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.988676+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:25.988676+0000","last_peered":"2024-09-15T03:06:25.988676+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:25.988676+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2483,"log_dups_size":0,"ondisk_log_size":2483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20289138,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2481,"num_write_kb":19828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2507","reported_seq":2570,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.953767+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:25.953767+0000","last_peered":"2024-09-15T03:06:25.953767+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:25.953767+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493938,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":20028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2580","reported_seq":2637,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.298689+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:30.298689+0000","last_peered":"2024-09-15T03:06:30.298689+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:30.298689+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2580,"log_dups_size":0,"ondisk_log_size":2580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21064932,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":20593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2507","reported_seq":2589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:26.812018+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:26.812018+0000","last_peered":"2024-09-15T03:06:26.812018+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:26.812018+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":0,"ondisk_log_size":2507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20518514,"num_objects":2505,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":20038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":447}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2622","reported_seq":2680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.149042+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:30.149042+0000","last_peered":"2024-09-15T03:06:30.149042+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:30.149042+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21434795,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":20940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2494","reported_seq":2878,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.260956+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:30.260956+0000","last_peered":"2024-09-15T03:06:30.260956+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:30.260956+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2494,"log_dups_size":0,"ondisk_log_size":2494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20421433,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":674}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2487","reported_seq":2564,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:26.758784+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:26.758784+0000","last_peered":"2024-09-15T03:06:26.758784+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:26.758784+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2487,"log_dups_size":0,"ondisk_log_size":2487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20364089,"num_objects":2486,"num_object_clones":0,"num_object_copies":9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":19887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":447}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2545","reported_seq":2595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.148637+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:30.148637+0000","last_peered":"2024-09-15T03:06:30.148637+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:30.148637+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2545,"log_dups_size":0,"ondisk_log_size":2545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20813426,"num_objects":2541,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2543,"num_write_kb":20340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2463","reported_seq":2543,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:26.776131+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:26.776131+0000","last_peered":"2024-09-15T03:06:26.776131+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:26.776131+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2463,"log_dups_size":0,"ondisk_log_size":2463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20149874,"num_objects":2460,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":19685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":472}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2477","reported_seq":2540,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:25.957982+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:25.957982+0000","last_peered":"2024-09-15T03:06:25.957982+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:25.957982+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20248178,"num_objects":2472,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2475,"num_write_kb":19788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":328981052,"num_objects":40163,"num_object_clones":0,"num_object_copies":160652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40190,"num_write_kb":321402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":519241728,"data_stored":519241728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40217,"ondisk_log_size":40217,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472884,"kb_used_data":162540,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93898956,"statfs":{"total":96636764160,"available":96152530944,"internally_reserved":0,"allocated":166440960,"data_stored":166281982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":2.1379999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":17.542000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0419999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":2.0680000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":2.097}]}]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":467188,"kb_used_data":157356,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93904652,"statfs":{"total":96636764160,"available":96158363648,"internally_reserved":0,"allocated":161132544,"data_stored":160976712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":8.1440000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.29399999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329008,"kb_used_data":26984,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94042832,"statfs":{"total":96636764160,"available":96299859968,"internally_reserved":0,"allocated":27631616,"data_stored":27472638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.091000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462448,"kb_used_data":156328,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93909392,"statfs":{"total":96636764160,"available":96163217408,"internally_reserved":0,"allocated":160079872,"data_stored":159920894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.377}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":161132544,"data_stored":161132544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28024832,"data_stored":28024832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162394112,"data_stored":162394112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167690240,"data_stored":167690240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:32.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:33.362 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 57 20 42104 42084 5.76666 2.86719 0.0979926 0.0270709 2024-09-15T03:06:33.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:33.873 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:33.900 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-15T03:06:32.656984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":337582652,"num_objects":41213,"num_object_clones":0,"num_object_copies":164852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41240,"num_write_kb":329802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41267,"ondisk_log_size":41267,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1739220,"kb_used_data":527540,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375748140,"statfs":{"total":386547056640,"available":384766095360,"internally_reserved":0,"allocated":540200960,"data_stored":539568194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,9,3,7,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40820736,"num_objects":4983,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4983,"num_write_kb":39864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004464"},"pg_stats":[{"pgid":"1.f","version":"31'2479","reported_seq":2520,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.218787+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:30.218787+0000","last_peered":"2024-09-15T03:06:30.218787+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:30.218787+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2479,"log_dups_size":0,"ondisk_log_size":2479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20307968,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2479,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2536","reported_seq":2598,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.062013+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:31.062013+0000","last_peered":"2024-09-15T03:06:31.062013+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:31.062013+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20739698,"num_objects":2532,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2534,"num_write_kb":20268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2685","reported_seq":2766,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.749114+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:31.749114+0000","last_peered":"2024-09-15T03:06:31.749114+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:31.749114+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2685,"log_dups_size":0,"ondisk_log_size":2685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21969721,"num_objects":2682,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":21462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":580}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2468","reported_seq":2531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.998251+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:30.998251+0000","last_peered":"2024-09-15T03:06:30.998251+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:30.998251+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20182642,"num_objects":2464,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2466,"num_write_kb":19717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2612","reported_seq":2668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.237490+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:30.237490+0000","last_peered":"2024-09-15T03:06:30.237490+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:30.237490+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2612,"log_dups_size":0,"ondisk_log_size":2612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21370482,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2610,"num_write_kb":20877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2604","reported_seq":2648,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.223041+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:30.223041+0000","last_peered":"2024-09-15T03:06:30.223041+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:30.223041+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2604,"log_dups_size":0,"ondisk_log_size":2604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21331968,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2596","reported_seq":2661,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.059682+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:31.059682+0000","last_peered":"2024-09-15T03:06:31.059682+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:31.059682+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2596,"log_dups_size":0,"ondisk_log_size":2596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214834,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2594,"num_write_kb":20732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":538}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2629","reported_seq":2692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.055819+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:31.055819+0000","last_peered":"2024-09-15T03:06:31.055819+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:31.055819+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21493362,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":21004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2580","reported_seq":2637,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.298689+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:30.298689+0000","last_peered":"2024-09-15T03:06:30.298689+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:30.298689+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2580,"log_dups_size":0,"ondisk_log_size":2580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21064932,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":20593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2629","reported_seq":2712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.837135+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:31.837135+0000","last_peered":"2024-09-15T03:06:31.837135+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:31.837135+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21517938,"num_objects":2627,"num_object_clones":0,"num_object_copies":10508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":21014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2622","reported_seq":2680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.149042+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:30.149042+0000","last_peered":"2024-09-15T03:06:30.149042+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:30.149042+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21434795,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":20940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2494","reported_seq":2878,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.260956+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:30.260956+0000","last_peered":"2024-09-15T03:06:30.260956+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:30.260956+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2494,"log_dups_size":0,"ondisk_log_size":2494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20421433,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":674}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2617","reported_seq":2693,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.833093+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:31.833093+0000","last_peered":"2024-09-15T03:06:31.833093+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:31.833093+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21429049,"num_objects":2616,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2616,"num_write_kb":20927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2545","reported_seq":2595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.148637+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:30.148637+0000","last_peered":"2024-09-15T03:06:30.148637+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:30.148637+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2545,"log_dups_size":0,"ondisk_log_size":2545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20813426,"num_objects":2541,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2543,"num_write_kb":20340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2578","reported_seq":2659,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.865051+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:31.865051+0000","last_peered":"2024-09-15T03:06:31.865051+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:31.865051+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2578,"log_dups_size":0,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21091954,"num_objects":2575,"num_object_clones":0,"num_object_copies":10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":587}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2593","reported_seq":2656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.971634+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:30.971634+0000","last_peered":"2024-09-15T03:06:30.971634+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:30.971634+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21198450,"num_objects":2588,"num_object_clones":0,"num_object_copies":10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2591,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":337582652,"num_objects":41213,"num_object_clones":0,"num_object_copies":164852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41240,"num_write_kb":329802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":542941184,"data_stored":542941184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41267,"ondisk_log_size":41267,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472884,"kb_used_data":162540,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93898956,"statfs":{"total":96636764160,"available":96152530944,"internally_reserved":0,"allocated":166440960,"data_stored":166281982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":2.1379999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":17.542000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0419999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":2.0680000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":2.097}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463728,"kb_used_data":166376,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93908112,"statfs":{"total":96636764160,"available":96161906688,"internally_reserved":0,"allocated":170369024,"data_stored":170213192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":2.3620000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.48899999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340888,"kb_used_data":34704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94030952,"statfs":{"total":96636764160,"available":96287694848,"internally_reserved":0,"allocated":35536896,"data_stored":35377918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.091000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461720,"kb_used_data":163920,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93910120,"statfs":{"total":96636764160,"available":96163962880,"internally_reserved":0,"allocated":167854080,"data_stored":167695102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":168943616,"data_stored":168943616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36089856,"data_stored":36089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170217472,"data_stored":170217472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167690240,"data_stored":167690240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:33.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:06:33.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2479", 'reported_seq': 2520, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.218787+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:06:30.218787+0000', 'last_peered': '2024-09-15T03:06:30.218787+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:06:30.218787+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2479, 'log_dups_size': 0, 'ondisk_log_size': 2479, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20307968, 'num_objects': 2479, 'num_object_clones': 0, 'num_object_copies': 9916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2479, 'num_write_kb': 19832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 503}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:33.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:06:33.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2536", 'reported_seq': 2598, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:31.062013+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:06:31.062013+0000', 'last_peered': '2024-09-15T03:06:31.062013+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:06:31.062013+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2536, 'log_dups_size': 0, 'ondisk_log_size': 2536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20739698, 'num_objects': 2532, 'num_object_clones': 0, 'num_object_copies': 10128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2534, 'num_write_kb': 20268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:33.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:06:33.901 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2685", 'reported_seq': 2766, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:31.749114+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:06:31.749114+0000', 'last_peered': '2024-09-15T03:06:31.749114+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:06:31.749114+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2685, 'log_dups_size': 0, 'ondisk_log_size': 2685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21969721, 'num_objects': 2682, 'num_object_clones': 0, 'num_object_copies': 10728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2684, 'num_write_kb': 21462, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 580}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:33.901 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2468", 'reported_seq': 2531, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.998251+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:06:30.998251+0000', 'last_peered': '2024-09-15T03:06:30.998251+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:06:30.998251+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2468, 'log_dups_size': 0, 'ondisk_log_size': 2468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20182642, 'num_objects': 2464, 'num_object_clones': 0, 'num_object_copies': 9856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2466, 'num_write_kb': 19717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2612", 'reported_seq': 2668, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.237490+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:06:30.237490+0000', 'last_peered': '2024-09-15T03:06:30.237490+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:06:30.237490+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2612, 'log_dups_size': 0, 'ondisk_log_size': 2612, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21370482, 'num_objects': 2609, 'num_object_clones': 0, 'num_object_copies': 10436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2610, 'num_write_kb': 20877, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 502}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2604", 'reported_seq': 2648, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.223041+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:06:30.223041+0000', 'last_peered': '2024-09-15T03:06:30.223041+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:06:30.223041+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2604, 'log_dups_size': 0, 'ondisk_log_size': 2604, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21331968, 'num_objects': 2604, 'num_object_clones': 0, 'num_object_copies': 10416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2604, 'num_write_kb': 20832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 548}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2596", 'reported_seq': 2661, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:31.059682+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:06:31.059682+0000', 'last_peered': '2024-09-15T03:06:31.059682+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:06:31.059682+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2596, 'log_dups_size': 0, 'ondisk_log_size': 2596, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21214834, 'num_objects': 2590, 'num_object_clones': 0, 'num_object_copies': 10360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2594, 'num_write_kb': 20732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 538}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2629", 'reported_seq': 2692, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:31.055819+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:06:31.055819+0000', 'last_peered': '2024-09-15T03:06:31.055819+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:06:31.055819+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2629, 'log_dups_size': 0, 'ondisk_log_size': 2629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21493362, 'num_objects': 2624, 'num_object_clones': 0, 'num_object_copies': 10496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2627, 'num_write_kb': 21004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 543}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:33.902 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2580", 'reported_seq': 2637, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.298689+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:06:30.298689+0000', 'last_peered': '2024-09-15T03:06:30.298689+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:06:30.298689+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2580, 'log_dups_size': 0, 'ondisk_log_size': 2580, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21064932, 'num_objects': 2572, 'num_object_clones': 0, 'num_object_copies': 10288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2577, 'num_write_kb': 20593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 530}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2629", 'reported_seq': 2712, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:31.837135+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:06:31.837135+0000', 'last_peered': '2024-09-15T03:06:31.837135+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:06:31.837135+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2629, 'log_dups_size': 0, 'ondisk_log_size': 2629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21517938, 'num_objects': 2627, 'num_object_clones': 0, 'num_object_copies': 10508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2627, 'num_write_kb': 21014, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 569}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2622", 'reported_seq': 2680, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.149042+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:06:30.149042+0000', 'last_peered': '2024-09-15T03:06:30.149042+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:06:30.149042+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2622, 'log_dups_size': 0, 'ondisk_log_size': 2622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21434795, 'num_objects': 2617, 'num_object_clones': 0, 'num_object_copies': 10468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2619, 'num_write_kb': 20940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 539}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2494", 'reported_seq': 2878, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.260956+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:06:30.260956+0000', 'last_peered': '2024-09-15T03:06:30.260956+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:06:30.260956+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2494, 'log_dups_size': 0, 'ondisk_log_size': 2494, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20421433, 'num_objects': 2493, 'num_object_clones': 0, 'num_object_copies': 9972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1819, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2493, 'num_write_kb': 19943, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 174, 'num_bytes_recovered': 1417216, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 674}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2617", 'reported_seq': 2693, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:31.833093+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:06:31.833093+0000', 'last_peered': '2024-09-15T03:06:31.833093+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:06:31.833093+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2617, 'log_dups_size': 0, 'ondisk_log_size': 2617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21429049, 'num_objects': 2616, 'num_object_clones': 0, 'num_object_copies': 10464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2616, 'num_write_kb': 20927, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 577}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:06:33.903 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2545", 'reported_seq': 2595, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.148637+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:06:30.148637+0000', 'last_peered': '2024-09-15T03:06:30.148637+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:06:30.148637+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2545, 'log_dups_size': 0, 'ondisk_log_size': 2545, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20813426, 'num_objects': 2541, 'num_object_clones': 0, 'num_object_copies': 10164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2543, 'num_write_kb': 20340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:33.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:06:33.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2578", 'reported_seq': 2659, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:31.865051+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:06:31.865051+0000', 'last_peered': '2024-09-15T03:06:31.865051+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:06:31.865051+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2578, 'log_dups_size': 0, 'ondisk_log_size': 2578, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21091954, 'num_objects': 2575, 'num_object_clones': 0, 'num_object_copies': 10300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2576, 'num_write_kb': 20605, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 587}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:33.904 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:06:33.904 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2593", 'reported_seq': 2656, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:30.971634+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:06:30.971634+0000', 'last_peered': '2024-09-15T03:06:30.971634+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:06:30.971634+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2593, 'log_dups_size': 0, 'ondisk_log_size': 2593, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21198450, 'num_objects': 2588, 'num_object_clones': 0, 'num_object_copies': 10352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2591, 'num_write_kb': 20716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 506}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:33.904 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:06:34.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 58 20 42474 42454 5.71707 2.89062 0.0301942 0.0273087 2024-09-15T03:06:35.109 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:35.109 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32483/164852 objects degraded (19.704%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":118,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41213,"data_bytes":337582652,"bytes_used":1780961280,"bytes_avail":384766095360,"bytes_total":386547056640,"degraded_objects":32483,"degraded_total":164852,"degraded_ratio":0.19704340863319827,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":66876,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"write_bytes_sec":2648444,"read_op_per_sec":0,"write_op_per_sec":323},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T03:06:26.653935+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T03:06:35.148 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:06:35.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:35.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 59 20 42819 42799 5.66586 2.69531 0.0626129 0.0275579 2024-09-15T03:06:36.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:06:36.360275+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0277894 2024-09-15T03:06:36.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:06:36.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 60 19 43184 43165 5.61909 2.85938 0.0814959 0.0277894 2024-09-15T03:06:36.439 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:36.453 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:36.472 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-15T03:06:34.657429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":337582652,"num_objects":41213,"num_object_clones":0,"num_object_copies":164852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41240,"num_write_kb":329802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41267,"ondisk_log_size":41267,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1739220,"kb_used_data":527540,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375748140,"statfs":{"total":386547056640,"available":384766095360,"internally_reserved":0,"allocated":540200960,"data_stored":539568194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,9,3,7,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31793152,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3881,"num_write_kb":31048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004463"},"pg_stats":[{"pgid":"1.f","version":"31'2479","reported_seq":2520,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.218787+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:30.218787+0000","last_peered":"2024-09-15T03:06:30.218787+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:30.218787+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2479,"log_dups_size":0,"ondisk_log_size":2479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20307968,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2479,"num_write_kb":19832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2536","reported_seq":2598,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.062013+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:31.062013+0000","last_peered":"2024-09-15T03:06:31.062013+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:31.062013+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20739698,"num_objects":2532,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2534,"num_write_kb":20268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2685","reported_seq":2766,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.749114+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:31.749114+0000","last_peered":"2024-09-15T03:06:31.749114+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:31.749114+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2685,"log_dups_size":0,"ondisk_log_size":2685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21969721,"num_objects":2682,"num_object_clones":0,"num_object_copies":10728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":21462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":580}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2468","reported_seq":2531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.998251+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:30.998251+0000","last_peered":"2024-09-15T03:06:30.998251+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:30.998251+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20182642,"num_objects":2464,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2466,"num_write_kb":19717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2612","reported_seq":2668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.237490+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:30.237490+0000","last_peered":"2024-09-15T03:06:30.237490+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:30.237490+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2612,"log_dups_size":0,"ondisk_log_size":2612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21370482,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2610,"num_write_kb":20877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2604","reported_seq":2648,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.223041+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:30.223041+0000","last_peered":"2024-09-15T03:06:30.223041+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:30.223041+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2604,"log_dups_size":0,"ondisk_log_size":2604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21331968,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":20832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2596","reported_seq":2661,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.059682+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:31.059682+0000","last_peered":"2024-09-15T03:06:31.059682+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:31.059682+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2596,"log_dups_size":0,"ondisk_log_size":2596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21214834,"num_objects":2590,"num_object_clones":0,"num_object_copies":10360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2594,"num_write_kb":20732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":538}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2629","reported_seq":2692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.055819+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:31.055819+0000","last_peered":"2024-09-15T03:06:31.055819+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:31.055819+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21493362,"num_objects":2624,"num_object_clones":0,"num_object_copies":10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":21004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2580","reported_seq":2637,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.298689+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:30.298689+0000","last_peered":"2024-09-15T03:06:30.298689+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:30.298689+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2580,"log_dups_size":0,"ondisk_log_size":2580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21064932,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2577,"num_write_kb":20593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2629","reported_seq":2712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.837135+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:31.837135+0000","last_peered":"2024-09-15T03:06:31.837135+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:31.837135+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2629,"log_dups_size":0,"ondisk_log_size":2629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21517938,"num_objects":2627,"num_object_clones":0,"num_object_copies":10508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2627,"num_write_kb":21014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2622","reported_seq":2680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.149042+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:30.149042+0000","last_peered":"2024-09-15T03:06:30.149042+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:30.149042+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2622,"log_dups_size":0,"ondisk_log_size":2622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21434795,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2619,"num_write_kb":20940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2494","reported_seq":2878,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.260956+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:30.260956+0000","last_peered":"2024-09-15T03:06:30.260956+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:30.260956+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2494,"log_dups_size":0,"ondisk_log_size":2494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20421433,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1819,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":19943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":674}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2617","reported_seq":2693,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.833093+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:31.833093+0000","last_peered":"2024-09-15T03:06:31.833093+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:31.833093+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2617,"log_dups_size":0,"ondisk_log_size":2617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21429049,"num_objects":2616,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2616,"num_write_kb":20927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2545","reported_seq":2595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.148637+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:30.148637+0000","last_peered":"2024-09-15T03:06:30.148637+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:30.148637+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2545,"log_dups_size":0,"ondisk_log_size":2545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20813426,"num_objects":2541,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2543,"num_write_kb":20340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":505}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2578","reported_seq":2659,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:31.865051+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:31.865051+0000","last_peered":"2024-09-15T03:06:31.865051+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:31.865051+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2578,"log_dups_size":0,"ondisk_log_size":2578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21091954,"num_objects":2575,"num_object_clones":0,"num_object_copies":10300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2576,"num_write_kb":20605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":587}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2593","reported_seq":2656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:30.971634+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:30.971634+0000","last_peered":"2024-09-15T03:06:30.971634+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:30.971634+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2593,"log_dups_size":0,"ondisk_log_size":2593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21198450,"num_objects":2588,"num_object_clones":0,"num_object_copies":10352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2591,"num_write_kb":20716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":337582652,"num_objects":41213,"num_object_clones":0,"num_object_copies":164852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32483,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41240,"num_write_kb":329802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1417216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":542941184,"data_stored":542941184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41267,"ondisk_log_size":41267,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":472884,"kb_used_data":162540,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93898956,"statfs":{"total":96636764160,"available":96152530944,"internally_reserved":0,"allocated":166440960,"data_stored":166281982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,0,1,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":2.1379999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":17.542000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0419999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":2.0680000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":2.097}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463728,"kb_used_data":166376,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93908112,"statfs":{"total":96636764160,"available":96161906688,"internally_reserved":0,"allocated":170369024,"data_stored":170213192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":2.3620000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.48899999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":340888,"kb_used_data":34704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94030952,"statfs":{"total":96636764160,"available":96287694848,"internally_reserved":0,"allocated":35536896,"data_stored":35377918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.091000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461720,"kb_used_data":163920,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93910120,"statfs":{"total":96636764160,"available":96163962880,"internally_reserved":0,"allocated":167854080,"data_stored":167695102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.27300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":168943616,"data_stored":168943616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36089856,"data_stored":36089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170217472,"data_stored":170217472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167690240,"data_stored":167690240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:37.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 61 20 43571 43551 5.57641 3.01562 0.0742737 0.0280029 2024-09-15T03:06:38.361 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 62 20 43958 43938 5.53524 3.02344 0.0481012 0.0282142 2024-09-15T03:06:39.363 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 63 20 44335 44315 5.49398 2.94531 0.0943844 0.0284205 2024-09-15T03:06:39.474 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:40.363 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 64 20 44695 44675 5.45208 2.8125 0.065604 0.0286302 2024-09-15T03:06:40.792 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:40.800 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:40.858 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-15T03:06:40.658658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359619132,"num_objects":43903,"num_object_clones":0,"num_object_copies":175612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43930,"num_write_kb":351322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43957,"ondisk_log_size":43957,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1793024,"kb_used_data":564704,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375694336,"statfs":{"total":386547056640,"available":384711000064,"internally_reserved":0,"allocated":578256896,"data_stored":577624130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,5,10,9,5,9,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37535744,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4582,"num_write_kb":36656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002562"},"pg_stats":[{"pgid":"1.f","version":"31'2730","reported_seq":2772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.153054+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:40.153054+0000","last_peered":"2024-09-15T03:06:40.153054+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:40.153054+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22364160,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2640","reported_seq":2702,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.084803+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:36.084803+0000","last_peered":"2024-09-15T03:06:36.084803+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:36.084803+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2640,"log_dups_size":0,"ondisk_log_size":2640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21591666,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2638,"num_write_kb":21100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2803","reported_seq":2884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.862474+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:36.862474+0000","last_peered":"2024-09-15T03:06:36.862474+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:36.862474+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22936377,"num_objects":2800,"num_object_clones":0,"num_object_copies":11200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2802,"num_write_kb":22406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":698}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2572","reported_seq":2635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:35.932213+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:35.932213+0000","last_peered":"2024-09-15T03:06:35.932213+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:35.932213+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2572,"log_dups_size":0,"ondisk_log_size":2572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21034610,"num_objects":2568,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":625}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2843","reported_seq":2899,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.374887+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:40.374887+0000","last_peered":"2024-09-15T03:06:40.374887+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:40.374887+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23262834,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":22725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2850","reported_seq":2894,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.324684+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:40.324684+0000","last_peered":"2024-09-15T03:06:40.324684+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:40.324684+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23347200,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":22800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2729","reported_seq":2793,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.098686+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:36.098686+0000","last_peered":"2024-09-15T03:06:36.098686+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:36.098686+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22304370,"num_objects":2723,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2727,"num_write_kb":21796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2753","reported_seq":2816,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:35.995324+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:35.995324+0000","last_peered":"2024-09-15T03:06:35.995324+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:35.995324+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22509170,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":21996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2804","reported_seq":2861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.229001+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:40.229001+0000","last_peered":"2024-09-15T03:06:40.229001+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:40.229001+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2804,"log_dups_size":0,"ondisk_log_size":2804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22899940,"num_objects":2796,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":22385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2739","reported_seq":2822,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.826618+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:36.826618+0000","last_peered":"2024-09-15T03:06:36.826618+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:36.826618+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2739,"log_dups_size":0,"ondisk_log_size":2739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22419058,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2737,"num_write_kb":21894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2862","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.287242+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:40.287242+0000","last_peered":"2024-09-15T03:06:40.287242+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:40.287242+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23400875,"num_objects":2857,"num_object_clones":0,"num_object_copies":11428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2744","reported_seq":3282,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.381980+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:40.381980+0000","last_peered":"2024-09-15T03:06:40.381980+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:40.381980+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22469433,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2731","reported_seq":2807,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.818635+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:36.818635+0000","last_peered":"2024-09-15T03:06:36.818635+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:36.818635+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2731,"log_dups_size":0,"ondisk_log_size":2731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22362937,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":691}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2769","reported_seq":2818,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.374867+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:40.374867+0000","last_peered":"2024-09-15T03:06:40.374867+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:40.374867+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2769,"log_dups_size":0,"ondisk_log_size":2769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648434,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":22132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":729}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2697","reported_seq":2778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.811736+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:36.811736+0000","last_peered":"2024-09-15T03:06:36.811736+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:36.811736+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2697,"log_dups_size":0,"ondisk_log_size":2697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22066802,"num_objects":2694,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":21557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2691","reported_seq":2754,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:35.959343+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:35.959343+0000","last_peered":"2024-09-15T03:06:35.959343+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:35.959343+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22001266,"num_objects":2686,"num_object_clones":0,"num_object_copies":10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":359619132,"num_objects":43903,"num_object_clones":0,"num_object_copies":175612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43930,"num_write_kb":351322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581058560,"data_stored":581058560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43957,"ondisk_log_size":43957,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483888,"kb_used_data":177704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93887952,"statfs":{"total":96636764160,"available":96141262848,"internally_reserved":0,"allocated":181968896,"data_stored":181809918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,5,2,3,1,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8939999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.927},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.46400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479200,"kb_used_data":173528,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93892640,"statfs":{"total":96636764160,"available":96146063360,"internally_reserved":0,"allocated":177692672,"data_stored":177536840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":5.4589999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":5.0949999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1189999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":5.0549999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356788,"kb_used_data":42284,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94015052,"statfs":{"total":96636764160,"available":96271413248,"internally_reserved":0,"allocated":43298816,"data_stored":43139838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473148,"kb_used_data":171188,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93898692,"statfs":{"total":96636764160,"available":96152260608,"internally_reserved":0,"allocated":175296512,"data_stored":175137534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":2.9929999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.33900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.371},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176488448,"data_stored":176488448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43786240,"data_stored":43786240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":177692672,"data_stored":177692672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183091200,"data_stored":183091200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:40.859 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:41.364 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 65 20 45069 45049 5.41316 2.92188 0.031217 0.0288519 2024-09-15T03:06:41.997 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:42.011 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:42.067 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-15T03:06:40.658658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359619132,"num_objects":43903,"num_object_clones":0,"num_object_copies":175612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43930,"num_write_kb":351322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43957,"ondisk_log_size":43957,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1793024,"kb_used_data":564704,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375694336,"statfs":{"total":386547056640,"available":384711000064,"internally_reserved":0,"allocated":578256896,"data_stored":577624130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,5,10,9,5,9,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37535744,"num_objects":4582,"num_object_clones":0,"num_object_copies":18328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4582,"num_write_kb":36656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002562"},"pg_stats":[{"pgid":"1.f","version":"31'2730","reported_seq":2772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.153054+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:40.153054+0000","last_peered":"2024-09-15T03:06:40.153054+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:40.153054+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22364160,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2640","reported_seq":2702,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.084803+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:36.084803+0000","last_peered":"2024-09-15T03:06:36.084803+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:36.084803+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2640,"log_dups_size":0,"ondisk_log_size":2640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21591666,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2638,"num_write_kb":21100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2803","reported_seq":2884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.862474+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:36.862474+0000","last_peered":"2024-09-15T03:06:36.862474+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:36.862474+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22936377,"num_objects":2800,"num_object_clones":0,"num_object_copies":11200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2802,"num_write_kb":22406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":698}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2572","reported_seq":2635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:35.932213+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:35.932213+0000","last_peered":"2024-09-15T03:06:35.932213+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:35.932213+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2572,"log_dups_size":0,"ondisk_log_size":2572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21034610,"num_objects":2568,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2570,"num_write_kb":20549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":625}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2843","reported_seq":2899,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.374887+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:40.374887+0000","last_peered":"2024-09-15T03:06:40.374887+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:40.374887+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23262834,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":22725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2850","reported_seq":2894,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.324684+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:40.324684+0000","last_peered":"2024-09-15T03:06:40.324684+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:40.324684+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23347200,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":22800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2729","reported_seq":2793,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.098686+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:36.098686+0000","last_peered":"2024-09-15T03:06:36.098686+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:36.098686+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2729,"log_dups_size":0,"ondisk_log_size":2729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22304370,"num_objects":2723,"num_object_clones":0,"num_object_copies":10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2727,"num_write_kb":21796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2753","reported_seq":2816,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:35.995324+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:35.995324+0000","last_peered":"2024-09-15T03:06:35.995324+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:35.995324+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22509170,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":21996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2804","reported_seq":2861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.229001+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:40.229001+0000","last_peered":"2024-09-15T03:06:40.229001+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:40.229001+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2804,"log_dups_size":0,"ondisk_log_size":2804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22899940,"num_objects":2796,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":22385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2739","reported_seq":2822,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.826618+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:36.826618+0000","last_peered":"2024-09-15T03:06:36.826618+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:36.826618+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2739,"log_dups_size":0,"ondisk_log_size":2739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22419058,"num_objects":2737,"num_object_clones":0,"num_object_copies":10948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2737,"num_write_kb":21894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":679}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2862","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.287242+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:40.287242+0000","last_peered":"2024-09-15T03:06:40.287242+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:40.287242+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23400875,"num_objects":2857,"num_object_clones":0,"num_object_copies":11428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2744","reported_seq":3282,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.381980+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:40.381980+0000","last_peered":"2024-09-15T03:06:40.381980+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:40.381980+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22469433,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2731","reported_seq":2807,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.818635+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:36.818635+0000","last_peered":"2024-09-15T03:06:36.818635+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:36.818635+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2731,"log_dups_size":0,"ondisk_log_size":2731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22362937,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":691}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2769","reported_seq":2818,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.374867+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:40.374867+0000","last_peered":"2024-09-15T03:06:40.374867+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:40.374867+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2769,"log_dups_size":0,"ondisk_log_size":2769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648434,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":22132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":729}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2697","reported_seq":2778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:36.811736+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:36.811736+0000","last_peered":"2024-09-15T03:06:36.811736+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:36.811736+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2697,"log_dups_size":0,"ondisk_log_size":2697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22066802,"num_objects":2694,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":21557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2691","reported_seq":2754,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:35.959343+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:35.959343+0000","last_peered":"2024-09-15T03:06:35.959343+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:35.959343+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22001266,"num_objects":2686,"num_object_clones":0,"num_object_copies":10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":21500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":359619132,"num_objects":43903,"num_object_clones":0,"num_object_copies":175612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43930,"num_write_kb":351322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581058560,"data_stored":581058560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43957,"ondisk_log_size":43957,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483888,"kb_used_data":177704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93887952,"statfs":{"total":96636764160,"available":96141262848,"internally_reserved":0,"allocated":181968896,"data_stored":181809918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,5,2,3,1,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8939999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.927},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.46400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479200,"kb_used_data":173528,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93892640,"statfs":{"total":96636764160,"available":96146063360,"internally_reserved":0,"allocated":177692672,"data_stored":177536840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":5.4589999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":5.0949999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.1189999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":5.0549999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356788,"kb_used_data":42284,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":94015052,"statfs":{"total":96636764160,"available":96271413248,"internally_reserved":0,"allocated":43298816,"data_stored":43139838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473148,"kb_used_data":171188,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93898692,"statfs":{"total":96636764160,"available":96152260608,"internally_reserved":0,"allocated":175296512,"data_stored":175137534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":2.9929999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.33900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.371},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176488448,"data_stored":176488448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43786240,"data_stored":43786240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":177692672,"data_stored":177692672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183091200,"data_stored":183091200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:42.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:06:42.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2730", 'reported_seq': 2772, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:40.153054+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:06:40.153054+0000', 'last_peered': '2024-09-15T03:06:40.153054+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:06:40.153054+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2730, 'log_dups_size': 0, 'ondisk_log_size': 2730, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22364160, 'num_objects': 2730, 'num_object_clones': 0, 'num_object_copies': 10920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2730, 'num_write_kb': 21840, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 754}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:42.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:06:42.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2640", 'reported_seq': 2702, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:36.084803+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:06:36.084803+0000', 'last_peered': '2024-09-15T03:06:36.084803+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:06:36.084803+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2640, 'log_dups_size': 0, 'ondisk_log_size': 2640, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21591666, 'num_objects': 2636, 'num_object_clones': 0, 'num_object_copies': 10544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2638, 'num_write_kb': 21100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 638}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2803", 'reported_seq': 2884, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:36.862474+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:06:36.862474+0000', 'last_peered': '2024-09-15T03:06:36.862474+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:06:36.862474+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2803, 'log_dups_size': 0, 'ondisk_log_size': 2803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22936377, 'num_objects': 2800, 'num_object_clones': 0, 'num_object_copies': 11200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2802, 'num_write_kb': 22406, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 698}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2572", 'reported_seq': 2635, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:35.932213+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:06:35.932213+0000', 'last_peered': '2024-09-15T03:06:35.932213+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:06:35.932213+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2572, 'log_dups_size': 0, 'ondisk_log_size': 2572, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21034610, 'num_objects': 2568, 'num_object_clones': 0, 'num_object_copies': 10272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2570, 'num_write_kb': 20549, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 625}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2843", 'reported_seq': 2899, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:40.374887+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:06:40.374887+0000', 'last_peered': '2024-09-15T03:06:40.374887+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:06:40.374887+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2843, 'log_dups_size': 0, 'ondisk_log_size': 2843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23262834, 'num_objects': 2840, 'num_object_clones': 0, 'num_object_copies': 11360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2841, 'num_write_kb': 22725, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 733}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2850", 'reported_seq': 2894, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:40.324684+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:06:40.324684+0000', 'last_peered': '2024-09-15T03:06:40.324684+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:06:40.324684+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2850, 'log_dups_size': 0, 'ondisk_log_size': 2850, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23347200, 'num_objects': 2850, 'num_object_clones': 0, 'num_object_copies': 11400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2850, 'num_write_kb': 22800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 794}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:06:42.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2729", 'reported_seq': 2793, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:36.098686+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:06:36.098686+0000', 'last_peered': '2024-09-15T03:06:36.098686+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:06:36.098686+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2729, 'log_dups_size': 0, 'ondisk_log_size': 2729, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22304370, 'num_objects': 2723, 'num_object_clones': 0, 'num_object_copies': 10892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2727, 'num_write_kb': 21796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 671}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2753", 'reported_seq': 2816, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:35.995324+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:06:35.995324+0000', 'last_peered': '2024-09-15T03:06:35.995324+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:06:35.995324+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2753, 'log_dups_size': 0, 'ondisk_log_size': 2753, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22509170, 'num_objects': 2748, 'num_object_clones': 0, 'num_object_copies': 10992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2751, 'num_write_kb': 21996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2804", 'reported_seq': 2861, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:40.229001+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:06:40.229001+0000', 'last_peered': '2024-09-15T03:06:40.229001+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:06:40.229001+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2804, 'log_dups_size': 0, 'ondisk_log_size': 2804, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22899940, 'num_objects': 2796, 'num_object_clones': 0, 'num_object_copies': 11184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2801, 'num_write_kb': 22385, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 754}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2739", 'reported_seq': 2822, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:36.826618+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:06:36.826618+0000', 'last_peered': '2024-09-15T03:06:36.826618+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:06:36.826618+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2739, 'log_dups_size': 0, 'ondisk_log_size': 2739, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22419058, 'num_objects': 2737, 'num_object_clones': 0, 'num_object_copies': 10948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2737, 'num_write_kb': 21894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 679}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2862", 'reported_seq': 2920, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:40.287242+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:06:40.287242+0000', 'last_peered': '2024-09-15T03:06:40.287242+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:06:40.287242+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2862, 'log_dups_size': 0, 'ondisk_log_size': 2862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23400875, 'num_objects': 2857, 'num_object_clones': 0, 'num_object_copies': 11428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2859, 'num_write_kb': 22860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 779}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2744", 'reported_seq': 3282, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:40.381980+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:06:40.381980+0000', 'last_peered': '2024-09-15T03:06:40.381980+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:06:40.381980+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2744, 'log_dups_size': 0, 'ondisk_log_size': 2744, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22469433, 'num_objects': 2743, 'num_object_clones': 0, 'num_object_copies': 10972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1742, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2743, 'num_write_kb': 21943, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 251, 'num_bytes_recovered': 2048000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1001}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:42.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:06:42.071 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2731", 'reported_seq': 2807, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:36.818635+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:06:36.818635+0000', 'last_peered': '2024-09-15T03:06:36.818635+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:06:36.818635+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2731, 'log_dups_size': 0, 'ondisk_log_size': 2731, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22362937, 'num_objects': 2730, 'num_object_clones': 0, 'num_object_copies': 10920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2730, 'num_write_kb': 21839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:42.071 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:06:42.071 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2769", 'reported_seq': 2818, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:40.374867+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:06:40.374867+0000', 'last_peered': '2024-09-15T03:06:40.374867+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:06:40.374867+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2769, 'log_dups_size': 0, 'ondisk_log_size': 2769, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22648434, 'num_objects': 2765, 'num_object_clones': 0, 'num_object_copies': 11060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2767, 'num_write_kb': 22132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 729}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:42.071 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:06:42.071 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2697", 'reported_seq': 2778, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:36.811736+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:06:36.811736+0000', 'last_peered': '2024-09-15T03:06:36.811736+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:06:36.811736+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2697, 'log_dups_size': 0, 'ondisk_log_size': 2697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22066802, 'num_objects': 2694, 'num_object_clones': 0, 'num_object_copies': 10776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2695, 'num_write_kb': 21557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 706}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:42.071 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:06:42.071 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2691", 'reported_seq': 2754, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:35.959343+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:06:35.959343+0000', 'last_peered': '2024-09-15T03:06:35.959343+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:06:35.959343+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2691, 'log_dups_size': 0, 'ondisk_log_size': 2691, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22001266, 'num_objects': 2686, 'num_object_clones': 0, 'num_object_copies': 10744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2686, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2689, 'num_write_kb': 21500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 604}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:42.072 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:06:42.363 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 66 20 45428 45408 5.37364 2.80469 0.06296 0.0290621 2024-09-15T03:06:43.278 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:43.279 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32406/179768 objects degraded (18.027%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":126,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44942,"data_bytes":368130620,"bytes_used":1860931584,"bytes_avail":384686125056,"bytes_total":386547056640,"degraded_objects":32406,"degraded_total":179768,"degraded_ratio":0.18026567575986829,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":52554,"recovering_keys_per_sec":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"write_bytes_sec":3261763,"read_op_per_sec":0,"write_op_per_sec":398},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T03:06:26.653935+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T03:06:43.296 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:06:43.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:43.363 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 67 20 45780 45760 5.33449 2.75 0.0854749 0.0292757 2024-09-15T03:06:43.998 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:44.033 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:44.052 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-15T03:06:42.659118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":368130620,"num_objects":44942,"num_object_clones":0,"num_object_copies":179768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44969,"num_write_kb":359634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44996,"ondisk_log_size":44996,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1817316,"kb_used_data":588996,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375670044,"statfs":{"total":386547056640,"available":384686125056,"internally_reserved":0,"allocated":603131904,"data_stored":602499138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,4,9,11,13,8,5,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39149568,"num_objects":4779,"num_object_clones":0,"num_object_copies":19116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4779,"num_write_kb":38232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002575"},"pg_stats":[{"pgid":"1.f","version":"31'2730","reported_seq":2772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.153054+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:40.153054+0000","last_peered":"2024-09-15T03:06:40.153054+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:40.153054+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2730,"log_dups_size":0,"ondisk_log_size":2730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22364160,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2752","reported_seq":2814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.114593+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:41.114593+0000","last_peered":"2024-09-15T03:06:41.114593+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:41.114593+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2752,"log_dups_size":0,"ondisk_log_size":2752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22509170,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2750,"num_write_kb":21996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2933","reported_seq":3014,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.783276+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:41.783276+0000","last_peered":"2024-09-15T03:06:41.783276+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:41.783276+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24001337,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2932,"num_write_kb":23446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":828}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2660","reported_seq":2723,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.979847+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:40.979847+0000","last_peered":"2024-09-15T03:06:40.979847+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:40.979847+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2660,"log_dups_size":0,"ondisk_log_size":2660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21755506,"num_objects":2656,"num_object_clones":0,"num_object_copies":10624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2658,"num_write_kb":21253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2843","reported_seq":2899,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.374887+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:40.374887+0000","last_peered":"2024-09-15T03:06:40.374887+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:40.374887+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2843,"log_dups_size":0,"ondisk_log_size":2843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23262834,"num_objects":2840,"num_object_clones":0,"num_object_copies":11360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2841,"num_write_kb":22725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2850","reported_seq":2894,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.324684+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:40.324684+0000","last_peered":"2024-09-15T03:06:40.324684+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:40.324684+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2850,"log_dups_size":0,"ondisk_log_size":2850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23347200,"num_objects":2850,"num_object_clones":0,"num_object_copies":11400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2850,"num_write_kb":22800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2854","reported_seq":2919,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.072072+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:41.072072+0000","last_peered":"2024-09-15T03:06:41.072072+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:41.072072+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2854,"log_dups_size":0,"ondisk_log_size":2854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23328370,"num_objects":2848,"num_object_clones":0,"num_object_copies":11392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2852,"num_write_kb":22796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":796}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2877","reported_seq":2940,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.084809+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:41.084809+0000","last_peered":"2024-09-15T03:06:41.084809+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:41.084809+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2877,"log_dups_size":0,"ondisk_log_size":2877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23524978,"num_objects":2872,"num_object_clones":0,"num_object_copies":11488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2875,"num_write_kb":22988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2804","reported_seq":2861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.229001+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:40.229001+0000","last_peered":"2024-09-15T03:06:40.229001+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:40.229001+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2804,"log_dups_size":0,"ondisk_log_size":2804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22899940,"num_objects":2796,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":22385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2841","reported_seq":2924,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.626292+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:41.626292+0000","last_peered":"2024-09-15T03:06:41.626292+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:41.626292+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2841,"log_dups_size":0,"ondisk_log_size":2841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23254642,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2839,"num_write_kb":22710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2862","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.287242+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:40.287242+0000","last_peered":"2024-09-15T03:06:40.287242+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:40.287242+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23400875,"num_objects":2857,"num_object_clones":0,"num_object_copies":11428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":779}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2744","reported_seq":3282,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.381980+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:40.381980+0000","last_peered":"2024-09-15T03:06:40.381980+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:40.381980+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2744,"log_dups_size":0,"ondisk_log_size":2744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22469433,"num_objects":2743,"num_object_clones":0,"num_object_copies":10972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2743,"num_write_kb":21943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1001}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2828","reported_seq":2905,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.864434+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:41.864434+0000","last_peered":"2024-09-15T03:06:41.864434+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:41.864434+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23157561,"num_objects":2827,"num_object_clones":0,"num_object_copies":11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":22615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2769","reported_seq":2818,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:40.374867+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:40.374867+0000","last_peered":"2024-09-15T03:06:40.374867+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:40.374867+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2769,"log_dups_size":0,"ondisk_log_size":2769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648434,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2767,"num_write_kb":22132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":729}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2825","reported_seq":2906,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.754747+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:41.754747+0000","last_peered":"2024-09-15T03:06:41.754747+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:41.754747+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2825,"log_dups_size":0,"ondisk_log_size":2825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23115378,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":22581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2824","reported_seq":2887,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.122089+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:41.122089+0000","last_peered":"2024-09-15T03:06:41.122089+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:41.122089+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2824,"log_dups_size":0,"ondisk_log_size":2824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23090802,"num_objects":2819,"num_object_clones":0,"num_object_copies":11276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2822,"num_write_kb":22564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":368130620,"num_objects":44942,"num_object_clones":0,"num_object_copies":179768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44969,"num_write_kb":359634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":2048000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":604397568,"data_stored":604397568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44996,"ondisk_log_size":44996,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483888,"kb_used_data":177704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93887952,"statfs":{"total":96636764160,"available":96141262848,"internally_reserved":0,"allocated":181968896,"data_stored":181809918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,5,2,3,1,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.312}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8939999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":1.927},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.46400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490968,"kb_used_data":181136,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93880872,"statfs":{"total":96636764160,"available":96134012928,"internally_reserved":0,"allocated":185483264,"data_stored":185327432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":6.6920000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":5.0119999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352056,"kb_used_data":50032,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94019784,"statfs":{"total":96636764160,"available":96276258816,"internally_reserved":0,"allocated":51232768,"data_stored":51073790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,6,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.21}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3450000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490404,"kb_used_data":180124,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93881436,"statfs":{"total":96636764160,"available":96134590464,"internally_reserved":0,"allocated":184446976,"data_stored":184287998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,3,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.216},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.36499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184303616,"data_stored":184303616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51675136,"data_stored":51675136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":185327616,"data_stored":185327616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183091200,"data_stored":183091200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:44.363 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 68 20 46089 46069 5.29154 2.41406 0.0249094 0.0295081 2024-09-15T03:06:45.364 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 69 20 46456 46436 5.25641 2.86719 0.153958 0.0296969 2024-09-15T03:06:46.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 70 20 46827 46807 5.22262 2.89844 0.0673448 0.0298975 2024-09-15T03:06:47.054 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:47.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 71 20 47192 47172 5.18923 2.85156 0.0469481 0.0300909 2024-09-15T03:06:48.183 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:48.206 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:48.254 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-15T03:06:46.661477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":378960444,"num_objects":46264,"num_object_clones":0,"num_object_copies":185056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46291,"num_write_kb":370210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46318,"ondisk_log_size":46318,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1821984,"kb_used_data":610304,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375665376,"statfs":{"total":386547056640,"available":384681345024,"internally_reserved":0,"allocated":624951296,"data_stored":624318530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,9,16,5,5,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41377792,"num_objects":5051,"num_object_clones":0,"num_object_copies":20204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5051,"num_write_kb":40408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004047"},"pg_stats":[{"pgid":"1.f","version":"31'2859","reported_seq":2899,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.391013+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:45.391013+0000","last_peered":"2024-09-15T03:06:45.391013+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:45.391013+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2859,"log_dups_size":0,"ondisk_log_size":2859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23420928,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2858","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.984344+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:45.984344+0000","last_peered":"2024-09-15T03:06:45.984344+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:45.984344+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2858,"log_dups_size":0,"ondisk_log_size":2858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23377522,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":22844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":856}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2933","reported_seq":3014,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.783276+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:41.783276+0000","last_peered":"2024-09-15T03:06:41.783276+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:41.783276+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2933,"log_dups_size":0,"ondisk_log_size":2933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24001337,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2932,"num_write_kb":23446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":828}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2771","reported_seq":2834,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.990751+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:45.990751+0000","last_peered":"2024-09-15T03:06:45.990751+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:45.990751+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22664818,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2956","reported_seq":3012,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.321834+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:45.321834+0000","last_peered":"2024-09-15T03:06:45.321834+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:45.321834+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24188530,"num_objects":2953,"num_object_clones":0,"num_object_copies":11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":23629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2948","reported_seq":2992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.275408+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:45.275408+0000","last_peered":"2024-09-15T03:06:45.275408+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:45.275408+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24150016,"num_objects":2948,"num_object_clones":0,"num_object_copies":11792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":23584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2956","reported_seq":3021,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.119816+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:46.119816+0000","last_peered":"2024-09-15T03:06:46.119816+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:46.119816+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24163954,"num_objects":2950,"num_object_clones":0,"num_object_copies":11800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":23612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":898}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2997","reported_seq":3060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.072204+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:46.072204+0000","last_peered":"2024-09-15T03:06:46.072204+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:46.072204+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2997,"log_dups_size":0,"ondisk_log_size":2997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24508018,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":23948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2911","reported_seq":2968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.288507+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:45.288507+0000","last_peered":"2024-09-15T03:06:45.288507+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:45.288507+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2911,"log_dups_size":0,"ondisk_log_size":2911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23776484,"num_objects":2903,"num_object_clones":0,"num_object_copies":11612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":23241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2841","reported_seq":2924,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.626292+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:41.626292+0000","last_peered":"2024-09-15T03:06:41.626292+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:41.626292+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2841,"log_dups_size":0,"ondisk_log_size":2841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23254642,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2839,"num_write_kb":22710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":781}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2978","reported_seq":3035,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.314456+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:45.314456+0000","last_peered":"2024-09-15T03:06:45.314456+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:45.314456+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24351147,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2856","reported_seq":3460,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.353913+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:45.353913+0000","last_peered":"2024-09-15T03:06:45.353913+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:45.353913+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23386937,"num_objects":2855,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1709,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2855,"num_write_kb":22839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1146}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2828","reported_seq":2905,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.864434+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:41.864434+0000","last_peered":"2024-09-15T03:06:41.864434+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:41.864434+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2828,"log_dups_size":0,"ondisk_log_size":2828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23157561,"num_objects":2827,"num_object_clones":0,"num_object_copies":11308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2827,"num_write_kb":22615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2869","reported_seq":2919,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.209231+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:45.209231+0000","last_peered":"2024-09-15T03:06:45.209231+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:45.209231+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2869,"log_dups_size":0,"ondisk_log_size":2869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23467634,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":22932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":829}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2825","reported_seq":2906,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:41.754747+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:41.754747+0000","last_peered":"2024-09-15T03:06:41.754747+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:41.754747+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2825,"log_dups_size":0,"ondisk_log_size":2825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23115378,"num_objects":2822,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2823,"num_write_kb":22581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2932","reported_seq":2995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.179379+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:46.179379+0000","last_peered":"2024-09-15T03:06:46.179379+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:46.179379+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2932,"log_dups_size":0,"ondisk_log_size":2932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23975538,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2930,"num_write_kb":23428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":378960444,"num_objects":46264,"num_object_clones":0,"num_object_copies":185056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46291,"num_write_kb":370210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626249728,"data_stored":626249728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46318,"ondisk_log_size":46318,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483956,"kb_used_data":186092,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93887884,"statfs":{"total":96636764160,"available":96141193216,"internally_reserved":0,"allocated":190558208,"data_stored":190399230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,9,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":1.2310000000000001},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":4.0919999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9050000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":7.5419999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.251}]}]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490968,"kb_used_data":181136,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93880872,"statfs":{"total":96636764160,"available":96134012928,"internally_reserved":0,"allocated":185483264,"data_stored":185327432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.311}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":6.6920000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":5.0119999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363324,"kb_used_data":57140,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94008516,"statfs":{"total":96636764160,"available":96264720384,"internally_reserved":0,"allocated":58511360,"data_stored":58352382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,4,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483736,"kb_used_data":185936,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93888104,"statfs":{"total":96636764160,"available":96141418496,"internally_reserved":0,"allocated":190398464,"data_stored":190239486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191545344,"data_stored":191545344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59064320,"data_stored":59064320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":185327616,"data_stored":185327616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190312448,"data_stored":190312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:48.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:48.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 72 20 47548 47528 5.15578 2.78125 0.0162175 0.0302861 2024-09-15T03:06:49.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 73 20 47906 47886 5.12347 2.79688 0.016561 0.030482 2024-09-15T03:06:49.460 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:49.499 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:49.531 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2024-09-15T03:06:48.661934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":382589500,"num_objects":46707,"num_object_clones":0,"num_object_copies":186828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46734,"num_write_kb":373754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46761,"ondisk_log_size":46761,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1816680,"kb_used_data":617480,"kb_used_omap":14,"kb_used_meta":1199153,"kb_avail":375670680,"statfs":{"total":386547056640,"available":384686776320,"internally_reserved":0,"allocated":632299520,"data_stored":631666754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227932699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,11,18,8,8,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33701888,"num_objects":4114,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4114,"num_write_kb":32912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":598016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004103"},"pg_stats":[{"pgid":"1.f","version":"31'2859","reported_seq":2899,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.391013+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:45.391013+0000","last_peered":"2024-09-15T03:06:45.391013+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:45.391013+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2859,"log_dups_size":0,"ondisk_log_size":2859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23420928,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":22872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2858","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.984344+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:45.984344+0000","last_peered":"2024-09-15T03:06:45.984344+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:45.984344+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2858,"log_dups_size":0,"ondisk_log_size":2858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23377522,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":22844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":856}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3041","reported_seq":3122,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.904557+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:46.904557+0000","last_peered":"2024-09-15T03:06:46.904557+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:46.904557+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3041,"log_dups_size":0,"ondisk_log_size":3041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24886073,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":24310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2771","reported_seq":2834,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.990751+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:45.990751+0000","last_peered":"2024-09-15T03:06:45.990751+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:45.990751+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22664818,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2956","reported_seq":3012,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.321834+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:45.321834+0000","last_peered":"2024-09-15T03:06:45.321834+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:45.321834+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24188530,"num_objects":2953,"num_object_clones":0,"num_object_copies":11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":23629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2948","reported_seq":2992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.275408+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:45.275408+0000","last_peered":"2024-09-15T03:06:45.275408+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:45.275408+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24150016,"num_objects":2948,"num_object_clones":0,"num_object_copies":11792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":23584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":892}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2956","reported_seq":3021,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.119816+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:46.119816+0000","last_peered":"2024-09-15T03:06:46.119816+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:46.119816+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24163954,"num_objects":2950,"num_object_clones":0,"num_object_copies":11800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":23612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":898}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2997","reported_seq":3060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.072204+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:46.072204+0000","last_peered":"2024-09-15T03:06:46.072204+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:46.072204+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2997,"log_dups_size":0,"ondisk_log_size":2997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24508018,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":23948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2911","reported_seq":2968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.288507+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:45.288507+0000","last_peered":"2024-09-15T03:06:45.288507+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:45.288507+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2911,"log_dups_size":0,"ondisk_log_size":2911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23776484,"num_objects":2903,"num_object_clones":0,"num_object_copies":11612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":23241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2959","reported_seq":3042,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.907452+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:46.907452+0000","last_peered":"2024-09-15T03:06:46.907452+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:46.907452+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2959,"log_dups_size":0,"ondisk_log_size":2959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24221298,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":23654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":899}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2978","reported_seq":3035,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.314456+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:45.314456+0000","last_peered":"2024-09-15T03:06:45.314456+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:45.314456+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24351147,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2975,"num_write_kb":23788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2856","reported_seq":3460,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.353913+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:45.353913+0000","last_peered":"2024-09-15T03:06:45.353913+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:45.353913+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2856,"log_dups_size":0,"ondisk_log_size":2856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23386937,"num_objects":2855,"num_object_clones":0,"num_object_copies":11420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1709,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2855,"num_write_kb":22839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1146}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2946","reported_seq":3022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.878978+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:46.878978+0000","last_peered":"2024-09-15T03:06:46.878978+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:46.878978+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24124217,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":906}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2869","reported_seq":2919,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.209231+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:45.209231+0000","last_peered":"2024-09-15T03:06:45.209231+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:45.209231+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2869,"log_dups_size":0,"ondisk_log_size":2869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23467634,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":22932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":829}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2924","reported_seq":3005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.860102+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:46.860102+0000","last_peered":"2024-09-15T03:06:46.860102+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:46.860102+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2924,"log_dups_size":0,"ondisk_log_size":2924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23926386,"num_objects":2921,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":23373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2932","reported_seq":2995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.179379+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:46.179379+0000","last_peered":"2024-09-15T03:06:46.179379+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:46.179379+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2932,"log_dups_size":0,"ondisk_log_size":2932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23975538,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2930,"num_write_kb":23428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":382589500,"num_objects":46707,"num_object_clones":0,"num_object_copies":186828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46734,"num_write_kb":373754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":633556992,"data_stored":633556992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46761,"ondisk_log_size":46761,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483956,"kb_used_data":186092,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93887884,"statfs":{"total":96636764160,"available":96141193216,"internally_reserved":0,"allocated":190558208,"data_stored":190399230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,9,1,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":1.2310000000000001},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":4.0919999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.9050000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":7.5419999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.251}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485664,"kb_used_data":188312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93886176,"statfs":{"total":96636764160,"available":96139444224,"internally_reserved":0,"allocated":192831488,"data_stored":192675656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.315}]}]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363324,"kb_used_data":57140,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94008516,"statfs":{"total":96636764160,"available":96264720384,"internally_reserved":0,"allocated":58511360,"data_stored":58352382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,4,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483736,"kb_used_data":185936,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93888104,"statfs":{"total":96636764160,"available":96141418496,"internally_reserved":0,"allocated":190398464,"data_stored":190239486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191545344,"data_stored":191545344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59064320,"data_stored":59064320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192634880,"data_stored":192634880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":190312448,"data_stored":190312448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:49.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:06:49.532 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2859", 'reported_seq': 2899, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.391013+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:06:45.391013+0000', 'last_peered': '2024-09-15T03:06:45.391013+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:06:45.391013+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2859, 'log_dups_size': 0, 'ondisk_log_size': 2859, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23420928, 'num_objects': 2859, 'num_object_clones': 0, 'num_object_copies': 11436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2859, 'num_write_kb': 22872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 883}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:49.532 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2858", 'reported_seq': 2920, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.984344+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:06:45.984344+0000', 'last_peered': '2024-09-15T03:06:45.984344+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:06:45.984344+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2858, 'log_dups_size': 0, 'ondisk_log_size': 2858, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23377522, 'num_objects': 2854, 'num_object_clones': 0, 'num_object_copies': 11416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2856, 'num_write_kb': 22844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 856}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3041", 'reported_seq': 3122, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:46.904557+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:06:46.904557+0000', 'last_peered': '2024-09-15T03:06:46.904557+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:06:46.904557+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3041, 'log_dups_size': 0, 'ondisk_log_size': 3041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24886073, 'num_objects': 3038, 'num_object_clones': 0, 'num_object_copies': 12152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3040, 'num_write_kb': 24310, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 936}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2771", 'reported_seq': 2834, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.990751+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:06:45.990751+0000', 'last_peered': '2024-09-15T03:06:45.990751+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:06:45.990751+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2771, 'log_dups_size': 0, 'ondisk_log_size': 2771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22664818, 'num_objects': 2767, 'num_object_clones': 0, 'num_object_copies': 11068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2769, 'num_write_kb': 22141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2956", 'reported_seq': 3012, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.321834+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:06:45.321834+0000', 'last_peered': '2024-09-15T03:06:45.321834+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:06:45.321834+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2956, 'log_dups_size': 0, 'ondisk_log_size': 2956, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24188530, 'num_objects': 2953, 'num_object_clones': 0, 'num_object_copies': 11812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2954, 'num_write_kb': 23629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 846}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2948", 'reported_seq': 2992, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.275408+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:06:45.275408+0000', 'last_peered': '2024-09-15T03:06:45.275408+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:06:45.275408+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2948, 'log_dups_size': 0, 'ondisk_log_size': 2948, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24150016, 'num_objects': 2948, 'num_object_clones': 0, 'num_object_copies': 11792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2948, 'num_write_kb': 23584, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 892}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:49.533 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2956", 'reported_seq': 3021, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:46.119816+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:06:46.119816+0000', 'last_peered': '2024-09-15T03:06:46.119816+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:06:46.119816+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2956, 'log_dups_size': 0, 'ondisk_log_size': 2956, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24163954, 'num_objects': 2950, 'num_object_clones': 0, 'num_object_copies': 11800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2954, 'num_write_kb': 23612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 898}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2997", 'reported_seq': 3060, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:46.072204+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:06:46.072204+0000', 'last_peered': '2024-09-15T03:06:46.072204+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:06:46.072204+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2997, 'log_dups_size': 0, 'ondisk_log_size': 2997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24508018, 'num_objects': 2992, 'num_object_clones': 0, 'num_object_copies': 11968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2995, 'num_write_kb': 23948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2911", 'reported_seq': 2968, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.288507+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:06:45.288507+0000', 'last_peered': '2024-09-15T03:06:45.288507+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:06:45.288507+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2911, 'log_dups_size': 0, 'ondisk_log_size': 2911, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23776484, 'num_objects': 2903, 'num_object_clones': 0, 'num_object_copies': 11612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2908, 'num_write_kb': 23241, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 861}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2959", 'reported_seq': 3042, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:46.907452+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:06:46.907452+0000', 'last_peered': '2024-09-15T03:06:46.907452+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:06:46.907452+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2959, 'log_dups_size': 0, 'ondisk_log_size': 2959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24221298, 'num_objects': 2957, 'num_object_clones': 0, 'num_object_copies': 11828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2957, 'num_write_kb': 23654, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 899}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2978", 'reported_seq': 3035, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.314456+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:06:45.314456+0000', 'last_peered': '2024-09-15T03:06:45.314456+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:06:45.314456+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2978, 'log_dups_size': 0, 'ondisk_log_size': 2978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24351147, 'num_objects': 2973, 'num_object_clones': 0, 'num_object_copies': 11892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2975, 'num_write_kb': 23788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 895}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:06:49.534 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2856", 'reported_seq': 3460, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.353913+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:06:45.353913+0000', 'last_peered': '2024-09-15T03:06:45.353913+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:06:45.353913+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2856, 'log_dups_size': 0, 'ondisk_log_size': 2856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23386937, 'num_objects': 2855, 'num_object_clones': 0, 'num_object_copies': 11420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1709, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2855, 'num_write_kb': 22839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 284, 'num_bytes_recovered': 2318336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1146}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:49.535 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:06:49.535 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2946", 'reported_seq': 3022, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:46.878978+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:06:46.878978+0000', 'last_peered': '2024-09-15T03:06:46.878978+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:06:46.878978+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2946, 'log_dups_size': 0, 'ondisk_log_size': 2946, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24124217, 'num_objects': 2945, 'num_object_clones': 0, 'num_object_copies': 11780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2945, 'num_write_kb': 23559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 906}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:49.535 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:06:49.535 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2869", 'reported_seq': 2919, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:45.209231+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:06:45.209231+0000', 'last_peered': '2024-09-15T03:06:45.209231+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:06:45.209231+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2869, 'log_dups_size': 0, 'ondisk_log_size': 2869, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23467634, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2867, 'num_write_kb': 22932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:49.535 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:06:49.535 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2924", 'reported_seq': 3005, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:46.860102+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:06:46.860102+0000', 'last_peered': '2024-09-15T03:06:46.860102+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:06:46.860102+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2924, 'log_dups_size': 0, 'ondisk_log_size': 2924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23926386, 'num_objects': 2921, 'num_object_clones': 0, 'num_object_copies': 11684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2921, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2922, 'num_write_kb': 23373, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 933}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:49.535 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:06:49.535 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2932", 'reported_seq': 2995, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:46.179379+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:06:46.179379+0000', 'last_peered': '2024-09-15T03:06:46.179379+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:06:46.179379+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2932, 'log_dups_size': 0, 'ondisk_log_size': 2932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23975538, 'num_objects': 2927, 'num_object_clones': 0, 'num_object_copies': 11708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2930, 'num_write_kb': 23428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 845}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:49.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:06:50.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 74 20 48231 48211 5.08855 2.53906 0.0506981 0.0306872 2024-09-15T03:06:51.114 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:51.114 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32336/190088 objects degraded (17.011%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":134,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47522,"data_bytes":389265980,"bytes_used":1876127744,"bytes_avail":384670928896,"bytes_total":386547056640,"degraded_objects":32336,"degraded_total":190088,"degraded_ratio":0.17011068557720635,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":75059,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":901120,"num_keys_recovered":0,"write_bytes_sec":3048781,"read_op_per_sec":0,"write_op_per_sec":372},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T03:06:26.653935+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T03:06:51.190 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:06:51.190 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:51.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 75 20 48583 48563 5.05737 2.75 0.0888422 0.0308767 2024-09-15T03:06:52.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:52.069 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:52.080 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-15T03:06:50.663639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389265980,"num_objects":47522,"num_object_clones":0,"num_object_copies":190088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47549,"num_write_kb":380274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":2621440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47576,"ondisk_log_size":47576,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1832156,"kb_used_data":624636,"kb_used_omap":14,"kb_used_meta":1207473,"kb_avail":375655204,"statfs":{"total":386547056640,"available":384670928896,"internally_reserved":0,"allocated":639627264,"data_stored":638994498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236452379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,5,5,12,9,7,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36601856,"num_objects":4468,"num_object_clones":0,"num_object_copies":17872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4468,"num_write_kb":35744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005406"},"pg_stats":[{"pgid":"1.f","version":"31'2965","reported_seq":3007,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.339486+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:50.339486+0000","last_peered":"2024-09-15T03:06:50.339486+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:50.339486+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2965,"log_dups_size":0,"ondisk_log_size":2965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24289280,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":23720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2858","reported_seq":2920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.984344+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:45.984344+0000","last_peered":"2024-09-15T03:06:45.984344+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:45.984344+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2858,"log_dups_size":0,"ondisk_log_size":2858,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23377522,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":22844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":856}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3041","reported_seq":3122,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.904557+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:46.904557+0000","last_peered":"2024-09-15T03:06:46.904557+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:46.904557+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3041,"log_dups_size":0,"ondisk_log_size":3041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24886073,"num_objects":3038,"num_object_clones":0,"num_object_copies":12152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":24310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2771","reported_seq":2834,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:45.990751+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:45.990751+0000","last_peered":"2024-09-15T03:06:45.990751+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:45.990751+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22664818,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3070","reported_seq":3124,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.399388+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:50.399388+0000","last_peered":"2024-09-15T03:06:50.399388+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:50.399388+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3070,"log_dups_size":0,"ondisk_log_size":3070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122418,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":24541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3081","reported_seq":3125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.415204+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:50.415204+0000","last_peered":"2024-09-15T03:06:50.415204+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:50.415204+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25239552,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2956","reported_seq":3021,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.119816+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:46.119816+0000","last_peered":"2024-09-15T03:06:46.119816+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:46.119816+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2956,"log_dups_size":0,"ondisk_log_size":2956,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24163954,"num_objects":2950,"num_object_clones":0,"num_object_copies":11800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2954,"num_write_kb":23612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":898}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2997","reported_seq":3060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.072204+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:46.072204+0000","last_peered":"2024-09-15T03:06:46.072204+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:46.072204+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2997,"log_dups_size":0,"ondisk_log_size":2997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24508018,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":23948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3003","reported_seq":3060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.225273+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:50.225273+0000","last_peered":"2024-09-15T03:06:50.225273+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:50.225273+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3003,"log_dups_size":0,"ondisk_log_size":3003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24530148,"num_objects":2995,"num_object_clones":0,"num_object_copies":11980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3000,"num_write_kb":23977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2959","reported_seq":3042,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.907452+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:46.907452+0000","last_peered":"2024-09-15T03:06:46.907452+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:46.907452+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2959,"log_dups_size":0,"ondisk_log_size":2959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24221298,"num_objects":2957,"num_object_clones":0,"num_object_copies":11828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":23654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":899}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3107","reported_seq":3164,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.422927+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:50.422927+0000","last_peered":"2024-09-15T03:06:50.422927+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:50.422927+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25407915,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3104,"num_write_kb":24820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2988","reported_seq":3665,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.417507+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:50.417507+0000","last_peered":"2024-09-15T03:06:50.417507+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:50.417507+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2988,"log_dups_size":0,"ondisk_log_size":2988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24468281,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":2621440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1315}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2946","reported_seq":3022,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.878978+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:46.878978+0000","last_peered":"2024-09-15T03:06:46.878978+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:46.878978+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24124217,"num_objects":2945,"num_object_clones":0,"num_object_copies":11780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":906}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2978","reported_seq":3028,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.357753+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:50.357753+0000","last_peered":"2024-09-15T03:06:50.357753+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:50.357753+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360562,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2924","reported_seq":3005,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.860102+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:46.860102+0000","last_peered":"2024-09-15T03:06:46.860102+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:46.860102+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2924,"log_dups_size":0,"ondisk_log_size":2924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23926386,"num_objects":2921,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":23373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2932","reported_seq":2995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:46.179379+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:46.179379+0000","last_peered":"2024-09-15T03:06:46.179379+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:46.179379+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2932,"log_dups_size":0,"ondisk_log_size":2932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23975538,"num_objects":2927,"num_object_clones":0,"num_object_copies":11708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2930,"num_write_kb":23428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389265980,"num_objects":47522,"num_object_clones":0,"num_object_copies":190088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47549,"num_write_kb":380274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":2621440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":640933888,"data_stored":640933888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47576,"ondisk_log_size":47576,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499432,"kb_used_data":193248,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93872408,"statfs":{"total":96636764160,"available":96125345792,"internally_reserved":0,"allocated":197885952,"data_stored":197726974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,3,2,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.63},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.39500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485664,"kb_used_data":188312,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93886176,"statfs":{"total":96636764160,"available":96139444224,"internally_reserved":0,"allocated":192831488,"data_stored":192675656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,3,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.247}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.315}]}]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":363324,"kb_used_data":57140,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":94008516,"statfs":{"total":96636764160,"available":96264720384,"internally_reserved":0,"allocated":58511360,"data_stored":58352382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,4,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.667}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483736,"kb_used_data":185936,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93888104,"statfs":{"total":96636764160,"available":96141418496,"internally_reserved":0,"allocated":190398464,"data_stored":190239486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191545344,"data_stored":191545344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59064320,"data_stored":59064320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192634880,"data_stored":192634880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":197689344,"data_stored":197689344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:52.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 76 20 48920 48900 5.02547 2.63281 0.038154 0.0310747 2024-09-15T03:06:53.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 77 20 49261 49241 4.99481 2.66406 0.028864 0.031269 2024-09-15T03:06:54.366 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 78 20 49613 49593 4.96603 2.75 0.0640625 0.0314434 2024-09-15T03:06:55.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:55.366 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 79 20 49978 49958 4.93926 2.85156 0.0586964 0.0316123 2024-09-15T03:06:56.277 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:56.321 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:56.346 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-15T03:06:54.664592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397146684,"num_objects":48484,"num_object_clones":0,"num_object_copies":193936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48511,"num_write_kb":387970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":2621440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48538,"ondisk_log_size":48538,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1857496,"kb_used_data":645816,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375629864,"statfs":{"total":386547056640,"available":384644980736,"internally_reserved":0,"allocated":661315584,"data_stored":660682818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,2,5,3,6,10,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29016064,"num_objects":3542,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-70,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3542,"num_write_kb":28336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005473"},"pg_stats":[{"pgid":"1.f","version":"31'2965","reported_seq":3007,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.339486+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:50.339486+0000","last_peered":"2024-09-15T03:06:50.339486+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:50.339486+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2965,"log_dups_size":0,"ondisk_log_size":2965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24289280,"num_objects":2965,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":23720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2948","reported_seq":3010,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.125996+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:51.125996+0000","last_peered":"2024-09-15T03:06:51.125996+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:51.125996+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2948,"log_dups_size":0,"ondisk_log_size":2948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114802,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2946,"num_write_kb":23564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3134","reported_seq":3215,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.857211+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:51.857211+0000","last_peered":"2024-09-15T03:06:51.857211+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:51.857211+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25647929,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":25054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1029}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2871","reported_seq":2934,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.967131+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:50.967131+0000","last_peered":"2024-09-15T03:06:50.967131+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:50.967131+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2871,"log_dups_size":0,"ondisk_log_size":2871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23484018,"num_objects":2867,"num_object_clones":0,"num_object_copies":11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":22941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3070","reported_seq":3124,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.399388+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:50.399388+0000","last_peered":"2024-09-15T03:06:50.399388+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:50.399388+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3070,"log_dups_size":0,"ondisk_log_size":3070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122418,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":24541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3081","reported_seq":3125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.415204+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:50.415204+0000","last_peered":"2024-09-15T03:06:50.415204+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:50.415204+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25239552,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1025}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3067","reported_seq":3132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.011041+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:51.011041+0000","last_peered":"2024-09-15T03:06:51.011041+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:51.011041+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3067,"log_dups_size":0,"ondisk_log_size":3067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25073266,"num_objects":3061,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3065,"num_write_kb":24500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1009}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3109","reported_seq":3172,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.043308+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:51.043308+0000","last_peered":"2024-09-15T03:06:51.043308+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:51.043308+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25425522,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":24844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1023}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3003","reported_seq":3060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.225273+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:50.225273+0000","last_peered":"2024-09-15T03:06:50.225273+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:50.225273+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3003,"log_dups_size":0,"ondisk_log_size":3003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24530148,"num_objects":2995,"num_object_clones":0,"num_object_copies":11980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3000,"num_write_kb":23977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3072","reported_seq":3155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.916940+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:51.916940+0000","last_peered":"2024-09-15T03:06:51.916940+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:51.916940+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3072,"log_dups_size":0,"ondisk_log_size":3072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25146994,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3070,"num_write_kb":24558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3107","reported_seq":3164,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.422927+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:50.422927+0000","last_peered":"2024-09-15T03:06:50.422927+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:50.422927+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25407915,"num_objects":3102,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3104,"num_write_kb":24820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2988","reported_seq":3665,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.417507+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:50.417507+0000","last_peered":"2024-09-15T03:06:50.417507+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:50.417507+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2988,"log_dups_size":0,"ondisk_log_size":2988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24468281,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1672,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":2621440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1315}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3049","reported_seq":3126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.894499+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:51.894499+0000","last_peered":"2024-09-15T03:06:51.894499+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:51.894499+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3049,"log_dups_size":0,"ondisk_log_size":3049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24967993,"num_objects":3048,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":24383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1009}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2978","reported_seq":3028,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:50.357753+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:50.357753+0000","last_peered":"2024-09-15T03:06:50.357753+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:50.357753+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2978,"log_dups_size":0,"ondisk_log_size":2978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24360562,"num_objects":2974,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2976,"num_write_kb":23804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":938}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3042","reported_seq":3123,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.860161+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:51.860161+0000","last_peered":"2024-09-15T03:06:51.860161+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:51.860161+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3042,"log_dups_size":0,"ondisk_log_size":3042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24893042,"num_objects":3039,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":24317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1051}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3054","reported_seq":3115,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.117539+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:51.117539+0000","last_peered":"2024-09-15T03:06:51.117539+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:51.117539+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3054,"log_dups_size":0,"ondisk_log_size":3054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24974962,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":24404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":967}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":397146684,"num_objects":48484,"num_object_clones":0,"num_object_copies":193936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48511,"num_write_kb":387970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":2621440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":662700032,"data_stored":662700032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48538,"ondisk_log_size":48538,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499432,"kb_used_data":193248,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93872408,"statfs":{"total":96636764160,"available":96125345792,"internally_reserved":0,"allocated":197885952,"data_stored":197726974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,3,2,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.42999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.63},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.39500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500924,"kb_used_data":195252,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93870916,"statfs":{"total":96636764160,"available":96123817984,"internally_reserved":0,"allocated":199938048,"data_stored":199782216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":5.4770000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":3.6800000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.434}]}]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":362120,"kb_used_data":64256,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":94009720,"statfs":{"total":96636764160,"available":96265953280,"internally_reserved":0,"allocated":65798144,"data_stored":65639166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.414999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":495020,"kb_used_data":193060,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93876820,"statfs":{"total":96636764160,"available":96129863680,"internally_reserved":0,"allocated":197693440,"data_stored":197534462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,1,1,3,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.26000000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.47799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198758400,"data_stored":198758400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66560000,"data_stored":66560000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199692288,"data_stored":199692288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":197689344,"data_stored":197689344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:56.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:56.365 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:06:56.365167+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0317956 2024-09-15T03:06:56.366 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:06:56.366 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 80 20 50323 50303 4.91122 2.69531 0.0663748 0.0317956 2024-09-15T03:06:57.366 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 81 20 50665 50645 4.88357 2.67188 0.0321008 0.0319756 2024-09-15T03:06:57.457 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:57.469 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:06:57.475 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-15T03:06:56.665023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":407878204,"num_objects":49794,"num_object_clones":0,"num_object_copies":199176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49821,"num_write_kb":398450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":2973696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49848,"ondisk_log_size":49848,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1889824,"kb_used_data":665664,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375597536,"statfs":{"total":386547056640,"available":384611876864,"internally_reserved":0,"allocated":681639936,"data_stored":681007170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,5,9,13,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39747584,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004052"},"pg_stats":[{"pgid":"1.f","version":"31'3081","reported_seq":3123,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.397019+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:55.397019+0000","last_peered":"2024-09-15T03:06:55.397019+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:55.397019+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25239552,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3043","reported_seq":3105,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.178513+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:56.178513+0000","last_peered":"2024-09-15T03:06:56.178513+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:56.178513+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3043,"log_dups_size":0,"ondisk_log_size":3043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24893042,"num_objects":3039,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":24324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3134","reported_seq":3215,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.857211+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:51.857211+0000","last_peered":"2024-09-15T03:06:51.857211+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:51.857211+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":0,"ondisk_log_size":3134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25647929,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":25054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1029}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2985","reported_seq":3047,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.179124+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:56.179124+0000","last_peered":"2024-09-15T03:06:56.179124+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:56.179124+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2985,"log_dups_size":0,"ondisk_log_size":2985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24417906,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3182","reported_seq":3238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.308717+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:55.308717+0000","last_peered":"2024-09-15T03:06:55.308717+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:55.308717+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3182,"log_dups_size":0,"ondisk_log_size":3182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26039922,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":25437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3191","reported_seq":3235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.409450+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:55.409450+0000","last_peered":"2024-09-15T03:06:55.409450+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:55.409450+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26140672,"num_objects":3191,"num_object_clones":0,"num_object_copies":12764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3191,"num_write_kb":25528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3162","reported_seq":3227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.139651+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:56.139651+0000","last_peered":"2024-09-15T03:06:56.139651+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:56.139651+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3162,"log_dups_size":0,"ondisk_log_size":3162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25851506,"num_objects":3156,"num_object_clones":0,"num_object_copies":12624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3160,"num_write_kb":25260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1104}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3214","reported_seq":3277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.189864+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:56.189864+0000","last_peered":"2024-09-15T03:06:56.189864+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:56.189864+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26285682,"num_objects":3209,"num_object_clones":0,"num_object_copies":12836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3212,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3114","reported_seq":3168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.426338+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:55.426338+0000","last_peered":"2024-09-15T03:06:55.426338+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:55.426338+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3114,"log_dups_size":0,"ondisk_log_size":3114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25439460,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3072","reported_seq":3155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.916940+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:51.916940+0000","last_peered":"2024-09-15T03:06:51.916940+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:51.916940+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3072,"log_dups_size":0,"ondisk_log_size":3072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25146994,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3070,"num_write_kb":24558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3232","reported_seq":3290,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.441754+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:55.441754+0000","last_peered":"2024-09-15T03:06:55.441754+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:55.441754+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3232,"log_dups_size":0,"ondisk_log_size":3232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26431915,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3229,"num_write_kb":25820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3108","reported_seq":3871,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.414977+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:55.414977+0000","last_peered":"2024-09-15T03:06:55.414977+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:55.414977+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3108,"log_dups_size":0,"ondisk_log_size":3108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25451321,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":24855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":2973696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3049","reported_seq":3126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.894499+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:51.894499+0000","last_peered":"2024-09-15T03:06:51.894499+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:51.894499+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3049,"log_dups_size":0,"ondisk_log_size":3049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24967993,"num_objects":3048,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":24383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1009}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3080","reported_seq":3130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.433431+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:55.433431+0000","last_peered":"2024-09-15T03:06:55.433431+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:55.433431+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3080,"log_dups_size":0,"ondisk_log_size":3080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25196146,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":24620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1040}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3042","reported_seq":3123,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:51.860161+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:51.860161+0000","last_peered":"2024-09-15T03:06:51.860161+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:51.860161+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3042,"log_dups_size":0,"ondisk_log_size":3042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24893042,"num_objects":3039,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3040,"num_write_kb":24317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1051}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3159","reported_seq":3222,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.171390+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:56.171390+0000","last_peered":"2024-09-15T03:06:56.171390+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:56.171390+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25835122,"num_objects":3154,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3157,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":407878204,"num_objects":49794,"num_object_clones":0,"num_object_copies":199176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49821,"num_write_kb":398450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":2973696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":684421120,"data_stored":684421120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49848,"ondisk_log_size":49848,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509172,"kb_used_data":198828,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93862668,"statfs":{"total":96636764160,"available":96115372032,"internally_reserved":0,"allocated":203599872,"data_stored":203440894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":2.0139999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0390000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":2.0710000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.47099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500924,"kb_used_data":195252,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93870916,"statfs":{"total":96636764160,"available":96123817984,"internally_reserved":0,"allocated":199938048,"data_stored":199782216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":5.4770000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.40500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":3.6800000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.434}]}]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373540,"kb_used_data":71516,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93998300,"statfs":{"total":96636764160,"available":96254259200,"internally_reserved":0,"allocated":73232384,"data_stored":73073406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.75}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506188,"kb_used_data":200068,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93865652,"statfs":{"total":96636764160,"available":96118427648,"internally_reserved":0,"allocated":204869632,"data_stored":204710654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.23499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205910016,"data_stored":205910016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73945088,"data_stored":73945088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199692288,"data_stored":199692288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204873728,"data_stored":204873728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:06:57.476 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:06:57.476 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3081", 'reported_seq': 3123, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:55.397019+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:06:55.397019+0000', 'last_peered': '2024-09-15T03:06:55.397019+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:06:55.397019+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3081, 'log_dups_size': 0, 'ondisk_log_size': 3081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25239552, 'num_objects': 3081, 'num_object_clones': 0, 'num_object_copies': 12324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3081, 'num_write_kb': 24648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1105}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:57.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:06:57.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3043", 'reported_seq': 3105, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:56.178513+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:06:56.178513+0000', 'last_peered': '2024-09-15T03:06:56.178513+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:06:56.178513+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3043, 'log_dups_size': 0, 'ondisk_log_size': 3043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24893042, 'num_objects': 3039, 'num_object_clones': 0, 'num_object_copies': 12156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3041, 'num_write_kb': 24324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1041}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:57.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:06:57.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3134", 'reported_seq': 3215, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:51.857211+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:06:51.857211+0000', 'last_peered': '2024-09-15T03:06:51.857211+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:06:51.857211+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3134, 'log_dups_size': 0, 'ondisk_log_size': 3134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25647929, 'num_objects': 3131, 'num_object_clones': 0, 'num_object_copies': 12524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3133, 'num_write_kb': 25054, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1029}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:57.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:06:57.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2985", 'reported_seq': 3047, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:56.179124+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:06:56.179124+0000', 'last_peered': '2024-09-15T03:06:56.179124+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:06:56.179124+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 2985, 'log_dups_size': 0, 'ondisk_log_size': 2985, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24417906, 'num_objects': 2981, 'num_object_clones': 0, 'num_object_copies': 11924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2983, 'num_write_kb': 23853, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:57.477 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:06:57.477 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3182", 'reported_seq': 3238, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:55.308717+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:06:55.308717+0000', 'last_peered': '2024-09-15T03:06:55.308717+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:06:55.308717+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3182, 'log_dups_size': 0, 'ondisk_log_size': 3182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26039922, 'num_objects': 3179, 'num_object_clones': 0, 'num_object_copies': 12716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3180, 'num_write_kb': 25437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1072}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:57.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:06:57.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3191", 'reported_seq': 3235, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:55.409450+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:06:55.409450+0000', 'last_peered': '2024-09-15T03:06:55.409450+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:06:55.409450+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3191, 'log_dups_size': 0, 'ondisk_log_size': 3191, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26140672, 'num_objects': 3191, 'num_object_clones': 0, 'num_object_copies': 12764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3191, 'num_write_kb': 25528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1135}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:57.481 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:06:57.481 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3162", 'reported_seq': 3227, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:56.139651+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:06:56.139651+0000', 'last_peered': '2024-09-15T03:06:56.139651+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:06:56.139651+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3162, 'log_dups_size': 0, 'ondisk_log_size': 3162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25851506, 'num_objects': 3156, 'num_object_clones': 0, 'num_object_copies': 12624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3160, 'num_write_kb': 25260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1104}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3214", 'reported_seq': 3277, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:56.189864+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:06:56.189864+0000', 'last_peered': '2024-09-15T03:06:56.189864+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:06:56.189864+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3214, 'log_dups_size': 0, 'ondisk_log_size': 3214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26285682, 'num_objects': 3209, 'num_object_clones': 0, 'num_object_copies': 12836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3212, 'num_write_kb': 25684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3114", 'reported_seq': 3168, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:55.426338+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:06:55.426338+0000', 'last_peered': '2024-09-15T03:06:55.426338+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:06:55.426338+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3114, 'log_dups_size': 0, 'ondisk_log_size': 3114, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25439460, 'num_objects': 3106, 'num_object_clones': 0, 'num_object_copies': 12424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3111, 'num_write_kb': 24865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1064}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3072", 'reported_seq': 3155, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:51.916940+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:06:51.916940+0000', 'last_peered': '2024-09-15T03:06:51.916940+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:06:51.916940+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3072, 'log_dups_size': 0, 'ondisk_log_size': 3072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25146994, 'num_objects': 3070, 'num_object_clones': 0, 'num_object_copies': 12280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3070, 'num_write_kb': 24558, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1012}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3232", 'reported_seq': 3290, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:55.441754+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:06:55.441754+0000', 'last_peered': '2024-09-15T03:06:55.441754+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:06:55.441754+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3232, 'log_dups_size': 0, 'ondisk_log_size': 3232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26431915, 'num_objects': 3227, 'num_object_clones': 0, 'num_object_copies': 12908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3229, 'num_write_kb': 25820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1149}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:06:57.482 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3108", 'reported_seq': 3871, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:55.414977+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:06:55.414977+0000', 'last_peered': '2024-09-15T03:06:55.414977+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:06:55.414977+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3108, 'log_dups_size': 0, 'ondisk_log_size': 3108, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25451321, 'num_objects': 3107, 'num_object_clones': 0, 'num_object_copies': 12428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1629, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3107, 'num_write_kb': 24855, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 364, 'num_bytes_recovered': 2973696, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:57.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:06:57.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3049", 'reported_seq': 3126, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:51.894499+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:06:51.894499+0000', 'last_peered': '2024-09-15T03:06:51.894499+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:06:51.894499+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3049, 'log_dups_size': 0, 'ondisk_log_size': 3049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24967993, 'num_objects': 3048, 'num_object_clones': 0, 'num_object_copies': 12192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3048, 'num_write_kb': 24383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1009}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:57.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:06:57.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3080", 'reported_seq': 3130, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:55.433431+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:06:55.433431+0000', 'last_peered': '2024-09-15T03:06:55.433431+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:06:55.433431+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3080, 'log_dups_size': 0, 'ondisk_log_size': 3080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25196146, 'num_objects': 3076, 'num_object_clones': 0, 'num_object_copies': 12304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3078, 'num_write_kb': 24620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1040}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:06:57.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:06:57.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3042", 'reported_seq': 3123, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:51.860161+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:06:51.860161+0000', 'last_peered': '2024-09-15T03:06:51.860161+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:06:51.860161+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3042, 'log_dups_size': 0, 'ondisk_log_size': 3042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24893042, 'num_objects': 3039, 'num_object_clones': 0, 'num_object_copies': 12156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3040, 'num_write_kb': 24317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1051}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:06:57.483 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:06:57.483 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3159", 'reported_seq': 3222, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:06:56.171390+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:06:56.171390+0000', 'last_peered': '2024-09-15T03:06:56.171390+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:06:56.171390+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3159, 'log_dups_size': 0, 'ondisk_log_size': 3159, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25835122, 'num_objects': 3154, 'num_object_clones': 0, 'num_object_copies': 12616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3157, 'num_write_kb': 25244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1072}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:06:57.484 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:06:58.370 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 82 20 50996 50976 4.85543 2.58594 0.0416289 0.0321583 2024-09-15T03:06:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:06:58.870 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32293/199176 objects degraded (16.213%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":141,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49794,"data_bytes":407878204,"bytes_used":1935179776,"bytes_avail":384611876864,"bytes_total":386547056640,"degraded_objects":32293,"degraded_total":199176,"degraded_ratio":0.16213298791018999,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":77115,"recovering_keys_per_sec":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"write_bytes_sec":3311180,"read_op_per_sec":0,"write_op_per_sec":404},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T03:06:26.653935+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T03:06:58.907 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:06:58.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:06:59.368 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 83 20 51337 51317 4.82903 2.66406 0.0591927 0.0323413 2024-09-15T03:07:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:00.048 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:00.074 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-15T03:06:58.667978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":411646524,"num_objects":50254,"num_object_clones":0,"num_object_copies":201016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50281,"num_write_kb":402130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":2973696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50308,"ondisk_log_size":50308,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1901008,"kb_used_data":672684,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375586352,"statfs":{"total":386547056640,"available":384600424448,"internally_reserved":0,"allocated":688828416,"data_stored":688195650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,7,14,14,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32686080,"num_objects":3990,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":31920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006501"},"pg_stats":[{"pgid":"1.f","version":"31'3081","reported_seq":3123,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.397019+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:06:55.397019+0000","last_peered":"2024-09-15T03:06:55.397019+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:06:55.397019+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25239552,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3043","reported_seq":3105,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.178513+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:06:56.178513+0000","last_peered":"2024-09-15T03:06:56.178513+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:06:56.178513+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3043,"log_dups_size":0,"ondisk_log_size":3043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24893042,"num_objects":3039,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":24324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3244","reported_seq":3325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.803383+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:06:56.803383+0000","last_peered":"2024-09-15T03:06:56.803383+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:06:56.803383+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26549049,"num_objects":3241,"num_object_clones":0,"num_object_copies":12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3243,"num_write_kb":25934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2985","reported_seq":3047,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.179124+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:06:56.179124+0000","last_peered":"2024-09-15T03:06:56.179124+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:06:56.179124+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":2985,"log_dups_size":0,"ondisk_log_size":2985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24417906,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2983,"num_write_kb":23853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3182","reported_seq":3238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.308717+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:06:55.308717+0000","last_peered":"2024-09-15T03:06:55.308717+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:06:55.308717+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3182,"log_dups_size":0,"ondisk_log_size":3182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26039922,"num_objects":3179,"num_object_clones":0,"num_object_copies":12716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3180,"num_write_kb":25437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1072}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3191","reported_seq":3235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.409450+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:06:55.409450+0000","last_peered":"2024-09-15T03:06:55.409450+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:06:55.409450+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26140672,"num_objects":3191,"num_object_clones":0,"num_object_copies":12764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3191,"num_write_kb":25528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1135}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3162","reported_seq":3227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.139651+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:06:56.139651+0000","last_peered":"2024-09-15T03:06:56.139651+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:06:56.139651+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3162,"log_dups_size":0,"ondisk_log_size":3162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25851506,"num_objects":3156,"num_object_clones":0,"num_object_copies":12624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3160,"num_write_kb":25260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1104}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3214","reported_seq":3277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.189864+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:06:56.189864+0000","last_peered":"2024-09-15T03:06:56.189864+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:06:56.189864+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26285682,"num_objects":3209,"num_object_clones":0,"num_object_copies":12836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3212,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3114","reported_seq":3168,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.426338+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:06:55.426338+0000","last_peered":"2024-09-15T03:06:55.426338+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:06:55.426338+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3114,"log_dups_size":0,"ondisk_log_size":3114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25439460,"num_objects":3106,"num_object_clones":0,"num_object_copies":12424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3183","reported_seq":3266,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.853192+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:06:56.853192+0000","last_peered":"2024-09-15T03:06:56.853192+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:06:56.853192+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3183,"log_dups_size":0,"ondisk_log_size":3183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26056306,"num_objects":3181,"num_object_clones":0,"num_object_copies":12724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3181,"num_write_kb":25446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1123}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3232","reported_seq":3290,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.441754+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:06:55.441754+0000","last_peered":"2024-09-15T03:06:55.441754+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:06:55.441754+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3232,"log_dups_size":0,"ondisk_log_size":3232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26431915,"num_objects":3227,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3229,"num_write_kb":25820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1149}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3108","reported_seq":3871,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.414977+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:06:55.414977+0000","last_peered":"2024-09-15T03:06:55.414977+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:06:55.414977+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3108,"log_dups_size":0,"ondisk_log_size":3108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25451321,"num_objects":3107,"num_object_clones":0,"num_object_copies":12428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":24855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":2973696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3170","reported_seq":3247,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.880880+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:06:56.880880+0000","last_peered":"2024-09-15T03:06:56.880880+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:06:56.880880+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25959225,"num_objects":3169,"num_object_clones":0,"num_object_copies":12676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3169,"num_write_kb":25351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1130}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3080","reported_seq":3130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:55.433431+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:06:55.433431+0000","last_peered":"2024-09-15T03:06:55.433431+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:06:55.433431+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3080,"log_dups_size":0,"ondisk_log_size":3080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25196146,"num_objects":3076,"num_object_clones":0,"num_object_copies":12304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":24620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1040}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3160","reported_seq":3240,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.899242+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:06:56.899242+0000","last_peered":"2024-09-15T03:06:56.899242+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:06:56.899242+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3160,"log_dups_size":0,"ondisk_log_size":3160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25859698,"num_objects":3157,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":25261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1169}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3159","reported_seq":3222,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:06:56.171390+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:06:56.171390+0000","last_peered":"2024-09-15T03:06:56.171390+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:06:56.171390+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25835122,"num_objects":3154,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3157,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":411646524,"num_objects":50254,"num_object_clones":0,"num_object_copies":201016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50281,"num_write_kb":402130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":2973696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":691650560,"data_stored":691650560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50308,"ondisk_log_size":50308,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509172,"kb_used_data":198828,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93862668,"statfs":{"total":96636764160,"available":96115372032,"internally_reserved":0,"allocated":203599872,"data_stored":203440894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,4,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":2.0139999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0390000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":2.0710000000000002},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.47099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512108,"kb_used_data":202272,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93859732,"statfs":{"total":96636764160,"available":96112365568,"internally_reserved":0,"allocated":207126528,"data_stored":206970696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":6.3949999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.32700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373540,"kb_used_data":71516,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93998300,"statfs":{"total":96636764160,"available":96254259200,"internally_reserved":0,"allocated":73232384,"data_stored":73073406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.75}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506188,"kb_used_data":200068,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93865652,"statfs":{"total":96636764160,"available":96118427648,"internally_reserved":0,"allocated":204869632,"data_stored":204710654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.23499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205910016,"data_stored":205910016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73945088,"data_stored":73945088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":206921728,"data_stored":206921728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204873728,"data_stored":204873728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:00.368 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 84 20 51680 51660 4.80345 2.67969 0.171934 0.0325042 2024-09-15T03:07:01.368 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 85 20 52031 52011 4.7792 2.74219 0.013798 0.0326758 2024-09-15T03:07:02.368 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 86 20 52354 52334 4.75297 2.52344 0.0549423 0.0328551 2024-09-15T03:07:03.076 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:03.368 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 87 20 52700 52680 4.72941 2.70312 0.0353953 0.0330119 2024-09-15T03:07:04.321 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:04.330 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:04.368 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 88 20 53034 53014 4.70532 2.60938 0.0369106 0.0331898 2024-09-15T03:07:04.429 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-15T03:07:02.669419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":425605692,"num_objects":51958,"num_object_clones":0,"num_object_copies":207832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51985,"num_write_kb":415762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3293184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52012,"ondisk_log_size":52012,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1906056,"kb_used_data":702696,"kb_used_omap":14,"kb_used_meta":1203313,"kb_avail":375581304,"statfs":{"total":386547056640,"available":384595255296,"internally_reserved":0,"allocated":719560704,"data_stored":718927938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232192539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,11,5,7,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36339712,"num_objects":4436,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4436,"num_write_kb":35488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005780"},"pg_stats":[{"pgid":"1.f","version":"31'3195","reported_seq":3237,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.413897+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:00.413897+0000","last_peered":"2024-09-15T03:07:00.413897+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:00.413897+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26173440,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":25560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3135","reported_seq":3196,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.201603+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:01.201603+0000","last_peered":"2024-09-15T03:07:01.201603+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:01.201603+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25646706,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":25060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3342","reported_seq":3422,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.935204+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:01.935204+0000","last_peered":"2024-09-15T03:07:01.935204+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:01.935204+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3342,"log_dups_size":0,"ondisk_log_size":3342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27351865,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":26718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3089","reported_seq":3151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.170898+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:01.170898+0000","last_peered":"2024-09-15T03:07:01.170898+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:01.170898+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25269874,"num_objects":3085,"num_object_clones":0,"num_object_copies":12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3087,"num_write_kb":24685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3292","reported_seq":3348,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.543784+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:00.543784+0000","last_peered":"2024-09-15T03:07:00.543784+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:00.543784+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26941042,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3290,"num_write_kb":26317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3307","reported_seq":3351,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.528502+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:00.528502+0000","last_peered":"2024-09-15T03:07:00.528502+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:00.528502+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27090944,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":26456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3264","reported_seq":3329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.114029+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:01.114029+0000","last_peered":"2024-09-15T03:07:01.114029+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:01.114029+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26687090,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1206}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3327","reported_seq":3390,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.152224+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:01.152224+0000","last_peered":"2024-09-15T03:07:01.152224+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:01.152224+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3327,"log_dups_size":0,"ondisk_log_size":3327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211378,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3229","reported_seq":3286,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.403484+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:00.403484+0000","last_peered":"2024-09-15T03:07:00.403484+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:00.403484+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26381540,"num_objects":3221,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3226,"num_write_kb":25785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3298","reported_seq":3380,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.843634+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:01.843634+0000","last_peered":"2024-09-15T03:07:01.843634+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:01.843634+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26998386,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3323","reported_seq":3381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.544590+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:00.544590+0000","last_peered":"2024-09-15T03:07:00.544590+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:00.544590+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3323,"log_dups_size":0,"ondisk_log_size":3323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27177387,"num_objects":3318,"num_object_clones":0,"num_object_copies":13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3220","reported_seq":4062,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.492871+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:00.492871+0000","last_peered":"2024-09-15T03:07:00.492871+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:00.492871+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26368825,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":25751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":3293184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3265","reported_seq":3342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.707806+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:01.707806+0000","last_peered":"2024-09-15T03:07:01.707806+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:01.707806+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26737465,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":26111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1225}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3190","reported_seq":3240,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.516526+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:00.516526+0000","last_peered":"2024-09-15T03:07:00.516526+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:00.516526+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3190,"log_dups_size":0,"ondisk_log_size":3190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26097266,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3262","reported_seq":3343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.738279+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:01.738279+0000","last_peered":"2024-09-15T03:07:01.738279+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:01.738279+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26695282,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":26077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1271}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3274","reported_seq":3337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.161913+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:01.161913+0000","last_peered":"2024-09-15T03:07:01.161913+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:01.161913+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26777202,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3272,"num_write_kb":26164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1187}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":425605692,"num_objects":51958,"num_object_clones":0,"num_object_copies":207832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51985,"num_write_kb":415762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3293184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":719896576,"data_stored":719896576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52012,"ondisk_log_size":52012,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504992,"kb_used_data":207128,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93866848,"statfs":{"total":96636764160,"available":96119652352,"internally_reserved":0,"allocated":212099072,"data_stored":211940094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,3,1,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":6.2089999999999996},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":4.3120000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":6.1609999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506268,"kb_used_data":208916,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93865572,"statfs":{"total":96636764160,"available":96118345728,"internally_reserved":0,"allocated":213929984,"data_stored":213774152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.946},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.20999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.36799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388832,"kb_used_data":78488,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93983008,"statfs":{"total":96636764160,"available":96238600192,"internally_reserved":0,"allocated":80371712,"data_stored":80212734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505964,"kb_used_data":208164,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93865876,"statfs":{"total":96636764160,"available":96118657024,"internally_reserved":0,"allocated":213159936,"data_stored":213000958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":12.170999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":14.882},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":11.6}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212951040,"data_stored":212951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81125376,"data_stored":81125376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":213831680,"data_stored":213831680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":211988480,"data_stored":211988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:04.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:05.368 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 89 20 53368 53348 4.68177 2.60938 0.101959 0.0333584 2024-09-15T03:07:05.849 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:05.864 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:05.876 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-15T03:07:04.669870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":425605692,"num_objects":51958,"num_object_clones":0,"num_object_copies":207832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51985,"num_write_kb":415762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3293184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52012,"ondisk_log_size":52012,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1906056,"kb_used_data":702696,"kb_used_omap":14,"kb_used_meta":1203313,"kb_avail":375581304,"statfs":{"total":386547056640,"available":384595255296,"internally_reserved":0,"allocated":719560704,"data_stored":718927938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232192539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,11,5,7,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28459008,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3474,"num_write_kb":27792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005772"},"pg_stats":[{"pgid":"1.f","version":"31'3195","reported_seq":3237,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.413897+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:00.413897+0000","last_peered":"2024-09-15T03:07:00.413897+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:00.413897+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3195,"log_dups_size":0,"ondisk_log_size":3195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26173440,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3195,"num_write_kb":25560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3135","reported_seq":3196,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.201603+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:01.201603+0000","last_peered":"2024-09-15T03:07:01.201603+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:01.201603+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25646706,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3133,"num_write_kb":25060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3342","reported_seq":3422,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.935204+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:01.935204+0000","last_peered":"2024-09-15T03:07:01.935204+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:01.935204+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3342,"log_dups_size":0,"ondisk_log_size":3342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27351865,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":26718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3089","reported_seq":3151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.170898+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:01.170898+0000","last_peered":"2024-09-15T03:07:01.170898+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:01.170898+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3089,"log_dups_size":0,"ondisk_log_size":3089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25269874,"num_objects":3085,"num_object_clones":0,"num_object_copies":12340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3087,"num_write_kb":24685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3292","reported_seq":3348,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.543784+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:00.543784+0000","last_peered":"2024-09-15T03:07:00.543784+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:00.543784+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3292,"log_dups_size":0,"ondisk_log_size":3292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26941042,"num_objects":3289,"num_object_clones":0,"num_object_copies":13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3290,"num_write_kb":26317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3307","reported_seq":3351,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.528502+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:00.528502+0000","last_peered":"2024-09-15T03:07:00.528502+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:00.528502+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3307,"log_dups_size":0,"ondisk_log_size":3307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27090944,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3307,"num_write_kb":26456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3264","reported_seq":3329,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.114029+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:01.114029+0000","last_peered":"2024-09-15T03:07:01.114029+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:01.114029+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26687090,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1206}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3327","reported_seq":3390,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.152224+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:01.152224+0000","last_peered":"2024-09-15T03:07:01.152224+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:01.152224+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3327,"log_dups_size":0,"ondisk_log_size":3327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211378,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3229","reported_seq":3286,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.403484+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:00.403484+0000","last_peered":"2024-09-15T03:07:00.403484+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:00.403484+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3229,"log_dups_size":0,"ondisk_log_size":3229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26381540,"num_objects":3221,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3226,"num_write_kb":25785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3298","reported_seq":3380,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.843634+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:01.843634+0000","last_peered":"2024-09-15T03:07:01.843634+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:01.843634+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26998386,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3323","reported_seq":3381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.544590+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:00.544590+0000","last_peered":"2024-09-15T03:07:00.544590+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:00.544590+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3323,"log_dups_size":0,"ondisk_log_size":3323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27177387,"num_objects":3318,"num_object_clones":0,"num_object_copies":13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3220","reported_seq":4062,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.492871+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:00.492871+0000","last_peered":"2024-09-15T03:07:00.492871+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:00.492871+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26368825,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3219,"num_write_kb":25751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":403,"num_bytes_recovered":3293184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1629}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3265","reported_seq":3342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.707806+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:01.707806+0000","last_peered":"2024-09-15T03:07:01.707806+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:01.707806+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26737465,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":26111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1225}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3190","reported_seq":3240,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:00.516526+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:00.516526+0000","last_peered":"2024-09-15T03:07:00.516526+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:00.516526+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3190,"log_dups_size":0,"ondisk_log_size":3190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26097266,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1150}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3262","reported_seq":3343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.738279+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:01.738279+0000","last_peered":"2024-09-15T03:07:01.738279+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:01.738279+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26695282,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":26077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1271}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3274","reported_seq":3337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.161913+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:01.161913+0000","last_peered":"2024-09-15T03:07:01.161913+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:01.161913+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3274,"log_dups_size":0,"ondisk_log_size":3274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26777202,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3272,"num_write_kb":26164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1187}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":425605692,"num_objects":51958,"num_object_clones":0,"num_object_copies":207832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51985,"num_write_kb":415762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3293184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":719896576,"data_stored":719896576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52012,"ondisk_log_size":52012,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504992,"kb_used_data":207128,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93866848,"statfs":{"total":96636764160,"available":96119652352,"internally_reserved":0,"allocated":212099072,"data_stored":211940094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,3,1,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":6.2089999999999996},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":4.3120000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":6.1609999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506268,"kb_used_data":208916,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93865572,"statfs":{"total":96636764160,"available":96118345728,"internally_reserved":0,"allocated":213929984,"data_stored":213774152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.946},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.20999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.36799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":388832,"kb_used_data":78488,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93983008,"statfs":{"total":96636764160,"available":96238600192,"internally_reserved":0,"allocated":80371712,"data_stored":80212734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.183}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505964,"kb_used_data":208164,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93865876,"statfs":{"total":96636764160,"available":96118657024,"internally_reserved":0,"allocated":213159936,"data_stored":213000958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":12.170999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":14.882},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":11.6}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212951040,"data_stored":212951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81125376,"data_stored":81125376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":213831680,"data_stored":213831680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":211988480,"data_stored":211988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:05.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:07:05.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3195", 'reported_seq': 3237, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:00.413897+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:07:00.413897+0000', 'last_peered': '2024-09-15T03:07:00.413897+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:07:00.413897+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3195, 'log_dups_size': 0, 'ondisk_log_size': 3195, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26173440, 'num_objects': 3195, 'num_object_clones': 0, 'num_object_copies': 12780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3195, 'num_write_kb': 25560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1219}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:05.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:07:05.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3135", 'reported_seq': 3196, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.201603+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:07:01.201603+0000', 'last_peered': '2024-09-15T03:07:01.201603+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:07:01.201603+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3135, 'log_dups_size': 0, 'ondisk_log_size': 3135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25646706, 'num_objects': 3131, 'num_object_clones': 0, 'num_object_copies': 12524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3133, 'num_write_kb': 25060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1133}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:05.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:07:05.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3342", 'reported_seq': 3422, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.935204+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:07:01.935204+0000', 'last_peered': '2024-09-15T03:07:01.935204+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:07:01.935204+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3342, 'log_dups_size': 0, 'ondisk_log_size': 3342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27351865, 'num_objects': 3339, 'num_object_clones': 0, 'num_object_copies': 13356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3341, 'num_write_kb': 26718, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1237}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:05.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:07:05.878 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3089", 'reported_seq': 3151, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.170898+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:07:01.170898+0000', 'last_peered': '2024-09-15T03:07:01.170898+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:07:01.170898+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3089, 'log_dups_size': 0, 'ondisk_log_size': 3089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25269874, 'num_objects': 3085, 'num_object_clones': 0, 'num_object_copies': 12340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3087, 'num_write_kb': 24685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1142}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:05.878 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3292", 'reported_seq': 3348, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:00.543784+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:07:00.543784+0000', 'last_peered': '2024-09-15T03:07:00.543784+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:07:00.543784+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3292, 'log_dups_size': 0, 'ondisk_log_size': 3292, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26941042, 'num_objects': 3289, 'num_object_clones': 0, 'num_object_copies': 13156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3289, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3290, 'num_write_kb': 26317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1182}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3307", 'reported_seq': 3351, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:00.528502+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:07:00.528502+0000', 'last_peered': '2024-09-15T03:07:00.528502+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:07:00.528502+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3307, 'log_dups_size': 0, 'ondisk_log_size': 3307, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27090944, 'num_objects': 3307, 'num_object_clones': 0, 'num_object_copies': 13228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3307, 'num_write_kb': 26456, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1251}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3264", 'reported_seq': 3329, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.114029+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:07:01.114029+0000', 'last_peered': '2024-09-15T03:07:01.114029+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:07:01.114029+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3264, 'log_dups_size': 0, 'ondisk_log_size': 3264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26687090, 'num_objects': 3258, 'num_object_clones': 0, 'num_object_copies': 13032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3262, 'num_write_kb': 26076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1206}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3327", 'reported_seq': 3390, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.152224+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:07:01.152224+0000', 'last_peered': '2024-09-15T03:07:01.152224+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:07:01.152224+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3327, 'log_dups_size': 0, 'ondisk_log_size': 3327, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27211378, 'num_objects': 3322, 'num_object_clones': 0, 'num_object_copies': 13288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3325, 'num_write_kb': 26588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3229", 'reported_seq': 3286, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:00.403484+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:07:00.403484+0000', 'last_peered': '2024-09-15T03:07:00.403484+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:07:00.403484+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3229, 'log_dups_size': 0, 'ondisk_log_size': 3229, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26381540, 'num_objects': 3221, 'num_object_clones': 0, 'num_object_copies': 12884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3226, 'num_write_kb': 25785, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1179}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:05.879 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3298", 'reported_seq': 3380, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.843634+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:07:01.843634+0000', 'last_peered': '2024-09-15T03:07:01.843634+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:07:01.843634+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3298, 'log_dups_size': 0, 'ondisk_log_size': 3298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26998386, 'num_objects': 3296, 'num_object_clones': 0, 'num_object_copies': 13184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3296, 'num_write_kb': 26366, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1238}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3323", 'reported_seq': 3381, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:00.544590+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:07:00.544590+0000', 'last_peered': '2024-09-15T03:07:00.544590+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:07:00.544590+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3323, 'log_dups_size': 0, 'ondisk_log_size': 3323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27177387, 'num_objects': 3318, 'num_object_clones': 0, 'num_object_copies': 13272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3320, 'num_write_kb': 26548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3220", 'reported_seq': 4062, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:00.492871+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:07:00.492871+0000', 'last_peered': '2024-09-15T03:07:00.492871+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:07:00.492871+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3220, 'log_dups_size': 0, 'ondisk_log_size': 3220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26368825, 'num_objects': 3219, 'num_object_clones': 0, 'num_object_copies': 12876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1590, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3219, 'num_write_kb': 25751, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 403, 'num_bytes_recovered': 3293184, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1629}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3265", 'reported_seq': 3342, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.707806+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:07:01.707806+0000', 'last_peered': '2024-09-15T03:07:01.707806+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:07:01.707806+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3265, 'log_dups_size': 0, 'ondisk_log_size': 3265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26737465, 'num_objects': 3264, 'num_object_clones': 0, 'num_object_copies': 13056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3264, 'num_write_kb': 26111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1225}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3190", 'reported_seq': 3240, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:00.516526+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:07:00.516526+0000', 'last_peered': '2024-09-15T03:07:00.516526+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:07:00.516526+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3190, 'log_dups_size': 0, 'ondisk_log_size': 3190, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26097266, 'num_objects': 3186, 'num_object_clones': 0, 'num_object_copies': 12744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3188, 'num_write_kb': 25500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:05.880 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:07:05.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3262", 'reported_seq': 3343, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.738279+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:07:01.738279+0000', 'last_peered': '2024-09-15T03:07:01.738279+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:07:01.738279+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3262, 'log_dups_size': 0, 'ondisk_log_size': 3262, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26695282, 'num_objects': 3259, 'num_object_clones': 0, 'num_object_copies': 13036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3260, 'num_write_kb': 26077, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1271}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:05.881 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:07:05.881 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3274", 'reported_seq': 3337, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:01.161913+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:07:01.161913+0000', 'last_peered': '2024-09-15T03:07:01.161913+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:07:01.161913+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3274, 'log_dups_size': 0, 'ondisk_log_size': 3274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26777202, 'num_objects': 3269, 'num_object_clones': 0, 'num_object_copies': 13076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3272, 'num_write_kb': 26164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1187}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:05.881 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:07:06.368 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 90 20 53681 53661 4.65692 2.44531 0.182199 0.0335289 2024-09-15T03:07:07.369 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 91 20 54004 53984 4.63347 2.52344 0.0459167 0.0337056 2024-09-15T03:07:07.442 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:07.442 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32219/212856 objects degraded (15.137%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":150,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":53214,"data_bytes":435894844,"bytes_used":1972576256,"bytes_avail":384574480384,"bytes_total":386547056640,"degraded_objects":32219,"degraded_total":212856,"degraded_ratio":0.15136524222948847,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":79834,"recovering_keys_per_sec":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"write_bytes_sec":3227478,"read_op_per_sec":0,"write_op_per_sec":393},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:07:07.498 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:07:07.499 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:08.369 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 92 20 54328 54308 4.61063 2.53125 0.0224291 0.0338704 2024-09-15T03:07:08.557 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:08.573 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:08.631 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-15T03:07:06.670297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":435894844,"num_objects":53214,"num_object_clones":0,"num_object_copies":212856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53241,"num_write_kb":425810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":3579904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53268,"ondisk_log_size":53268,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1926344,"kb_used_data":722984,"kb_used_omap":14,"kb_used_meta":1203313,"kb_avail":375561016,"statfs":{"total":386547056640,"available":384574480384,"internally_reserved":0,"allocated":740335616,"data_stored":739702850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232192539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,4,7,11,13,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38748160,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4730,"num_write_kb":37840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005705"},"pg_stats":[{"pgid":"1.f","version":"31'3314","reported_seq":3356,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:05.518342+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:05.518342+0000","last_peered":"2024-09-15T03:07:05.518342+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:05.518342+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3314,"log_dups_size":0,"ondisk_log_size":3314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27148288,"num_objects":3314,"num_object_clones":0,"num_object_copies":13256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3314,"num_write_kb":26512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3224","reported_seq":3286,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:06.094399+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:06.094399+0000","last_peered":"2024-09-15T03:07:06.094399+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:06.094399+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3224,"log_dups_size":0,"ondisk_log_size":3224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375794,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3222,"num_write_kb":25772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3342","reported_seq":3422,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.935204+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:01.935204+0000","last_peered":"2024-09-15T03:07:01.935204+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:01.935204+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3342,"log_dups_size":0,"ondisk_log_size":3342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27351865,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3341,"num_write_kb":26718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3184","reported_seq":3247,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:06.006049+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:06.006049+0000","last_peered":"2024-09-15T03:07:06.006049+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:06.006049+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3184,"log_dups_size":0,"ondisk_log_size":3184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26048114,"num_objects":3180,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3182,"num_write_kb":25445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1237}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3397","reported_seq":3453,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:05.522026+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:05.522026+0000","last_peered":"2024-09-15T03:07:05.522026+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:05.522026+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27801202,"num_objects":3394,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3395,"num_write_kb":27157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3424","reported_seq":3468,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:05.595988+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:05.595988+0000","last_peered":"2024-09-15T03:07:05.595988+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:05.595988+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3424,"log_dups_size":0,"ondisk_log_size":3424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28049408,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1368}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3369","reported_seq":3434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:06.244591+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:06.244591+0000","last_peered":"2024-09-15T03:07:06.244591+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:06.244591+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3369,"log_dups_size":0,"ondisk_log_size":3369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27547250,"num_objects":3363,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3367,"num_write_kb":26916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1311}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3430","reported_seq":3493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:06.182899+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:06.182899+0000","last_peered":"2024-09-15T03:07:06.182899+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:06.182899+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28055154,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3428,"num_write_kb":27412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3327","reported_seq":3384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:05.509965+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:05.509965+0000","last_peered":"2024-09-15T03:07:05.509965+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:05.509965+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3327,"log_dups_size":0,"ondisk_log_size":3327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27184356,"num_objects":3319,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3298","reported_seq":3380,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.843634+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:01.843634+0000","last_peered":"2024-09-15T03:07:01.843634+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:01.843634+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26998386,"num_objects":3296,"num_object_clones":0,"num_object_copies":13184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":26366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1238}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3406","reported_seq":3464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:05.533874+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:05.533874+0000","last_peered":"2024-09-15T03:07:05.533874+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:05.533874+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3406,"log_dups_size":0,"ondisk_log_size":3406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27857323,"num_objects":3401,"num_object_clones":0,"num_object_copies":13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3403,"num_write_kb":27212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3329","reported_seq":4240,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:05.568138+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:05.568138+0000","last_peered":"2024-09-15T03:07:05.568138+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:05.568138+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3329,"log_dups_size":0,"ondisk_log_size":3329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27261753,"num_objects":3328,"num_object_clones":0,"num_object_copies":13312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3328,"num_write_kb":26623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":438,"num_bytes_recovered":3579904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3265","reported_seq":3342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.707806+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:01.707806+0000","last_peered":"2024-09-15T03:07:01.707806+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:01.707806+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26737465,"num_objects":3264,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":26111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1225}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3311","reported_seq":3361,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:05.577759+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:05.577759+0000","last_peered":"2024-09-15T03:07:05.577759+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:05.577759+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3311,"log_dups_size":0,"ondisk_log_size":3311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27088498,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3309,"num_write_kb":26468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1271}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3262","reported_seq":3343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:01.738279+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:01.738279+0000","last_peered":"2024-09-15T03:07:01.738279+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:01.738279+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26695282,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":26077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1271}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3386","reported_seq":3448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:06.255215+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:06.255215+0000","last_peered":"2024-09-15T03:07:06.255215+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:06.255215+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3386,"log_dups_size":0,"ondisk_log_size":3386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27694706,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":435894844,"num_objects":53214,"num_object_clones":0,"num_object_copies":212856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53241,"num_write_kb":425810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":3579904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":740585472,"data_stored":740585472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53268,"ondisk_log_size":53268,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":520180,"kb_used_data":213996,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93851660,"statfs":{"total":96636764160,"available":96104099840,"internally_reserved":0,"allocated":219131904,"data_stored":218972926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.21299999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5609999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":3.5190000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":2.907}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506268,"kb_used_data":208916,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93865572,"statfs":{"total":96636764160,"available":96118345728,"internally_reserved":0,"allocated":213929984,"data_stored":213774152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.946},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.20999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.36799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":383240,"kb_used_data":85376,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93988600,"statfs":{"total":96636764160,"available":96244326400,"internally_reserved":0,"allocated":87425024,"data_stored":87266046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.217}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516656,"kb_used_data":214696,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93855184,"statfs":{"total":96636764160,"available":96107708416,"internally_reserved":0,"allocated":219848704,"data_stored":219689726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,7,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":2.5600000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":2.242}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.375}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":219738112,"data_stored":219738112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88125440,"data_stored":88125440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":213831680,"data_stored":213831680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":218890240,"data_stored":218890240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:09.369 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 93 20 54662 54642 4.58911 2.60938 0.0285193 0.0340298 2024-09-15T03:07:10.369 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 94 20 54995 54975 4.56796 2.60156 0.0426689 0.0341856 2024-09-15T03:07:11.369 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 95 20 55329 55309 4.54735 2.60938 0.0569964 0.0343446 2024-09-15T03:07:11.632 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:12.369 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 96 20 55658 55638 4.52674 2.57031 0.0969604 0.0345011 2024-09-15T03:07:12.871 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:12.892 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:12.900 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-15T03:07:12.672411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":452827708,"num_objects":55281,"num_object_clones":0,"num_object_copies":221124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55308,"num_write_kb":442346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55335,"ondisk_log_size":55335,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1988604,"kb_used_data":756124,"kb_used_omap":14,"kb_used_meta":1232433,"kb_avail":375498756,"statfs":{"total":386547056640,"available":384510726144,"internally_reserved":0,"allocated":774270976,"data_stored":773638210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1262011419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,4,5,6,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34889728,"num_objects":4259,"num_object_clones":0,"num_object_copies":17036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":34072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"1.f","version":"31'3394","reported_seq":3436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.460539+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:10.460539+0000","last_peered":"2024-09-15T03:07:10.460539+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:10.460539+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27803648,"num_objects":3394,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3394,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3326","reported_seq":3388,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.239571+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:11.239571+0000","last_peered":"2024-09-15T03:07:11.239571+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:11.239571+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3326,"log_dups_size":0,"ondisk_log_size":3326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211378,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3539","reported_seq":3620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.781234+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:11.781234+0000","last_peered":"2024-09-15T03:07:11.781234+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:11.781234+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28965689,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":28294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1434}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3272","reported_seq":3335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.169340+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:11.169340+0000","last_peered":"2024-09-15T03:07:11.169340+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:11.169340+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26769010,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3270,"num_write_kb":26149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3488","reported_seq":3544,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.514620+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:10.514620+0000","last_peered":"2024-09-15T03:07:10.514620+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:10.514620+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3488,"log_dups_size":0,"ondisk_log_size":3488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546674,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":27885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3542","reported_seq":3585,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.661336+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:10.661336+0000","last_peered":"2024-09-15T03:07:10.661336+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:10.661336+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29016064,"num_objects":3542,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3542,"num_write_kb":28336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3478","reported_seq":3543,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.124767+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:11.124767+0000","last_peered":"2024-09-15T03:07:11.124767+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:11.124767+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28440178,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3476,"num_write_kb":27788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3529","reported_seq":3592,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.877569+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:10.877569+0000","last_peered":"2024-09-15T03:07:10.877569+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:10.877569+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28866162,"num_objects":3524,"num_object_clones":0,"num_object_copies":14096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3440","reported_seq":3497,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.488483+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:10.488483+0000","last_peered":"2024-09-15T03:07:10.488483+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:10.488483+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3440,"log_dups_size":0,"ondisk_log_size":3440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28110052,"num_objects":3432,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3437,"num_write_kb":27473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3491","reported_seq":3574,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.936538+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:11.936538+0000","last_peered":"2024-09-15T03:07:11.936538+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:11.936538+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28579442,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3518","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.598432+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:10.598432+0000","last_peered":"2024-09-15T03:07:10.598432+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:10.598432+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28774827,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3443","reported_seq":4439,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.641676+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:10.641676+0000","last_peered":"2024-09-15T03:07:10.641676+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:10.641676+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3443,"log_dups_size":0,"ondisk_log_size":3443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28195641,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":27535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3485","reported_seq":3561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.958281+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:11.958281+0000","last_peered":"2024-09-15T03:07:11.958281+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:11.958281+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28539705,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":27871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3413","reported_seq":3463,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.597167+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:10.597167+0000","last_peered":"2024-09-15T03:07:10.597167+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:10.597167+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3413,"log_dups_size":0,"ondisk_log_size":3413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27924082,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1373}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3471","reported_seq":3552,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.948819+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:11.948819+0000","last_peered":"2024-09-15T03:07:11.948819+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:11.948819+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28407410,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3469,"num_write_kb":27749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3506","reported_seq":3567,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.244570+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:11.244570+0000","last_peered":"2024-09-15T03:07:11.244570+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:11.244570+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3506,"log_dups_size":0,"ondisk_log_size":3506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677746,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":28020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":452827708,"num_objects":55281,"num_object_clones":0,"num_object_copies":221124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55308,"num_write_kb":442346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":774656000,"data_stored":774656000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55335,"ondisk_log_size":55335,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530876,"kb_used_data":220532,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93840964,"statfs":{"total":96636764160,"available":96093147136,"internally_reserved":0,"allocated":225824768,"data_stored":225665790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,4,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.21199999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.318},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.36899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.39300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532028,"kb_used_data":222196,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93839812,"statfs":{"total":96636764160,"available":96091967488,"internally_reserved":0,"allocated":227528704,"data_stored":227372872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":6.6820000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.27100000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":7.6130000000000004},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":6.5919999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":9.1579999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.30499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394156,"kb_used_data":92132,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93977684,"statfs":{"total":96636764160,"available":96233148416,"internally_reserved":0,"allocated":94343168,"data_stored":94184190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":4.0019999999999998},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.23799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":3.7160000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":3.9369999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531544,"kb_used_data":221264,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93840296,"statfs":{"total":96636764160,"available":96092463104,"internally_reserved":0,"allocated":226574336,"data_stored":226415358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.495}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":2.0489999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.27900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.24399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226496512,"data_stored":226496512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95068160,"data_stored":95068160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227409920,"data_stored":227409920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225681408,"data_stored":225681408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:12.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:13.370 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 97 20 55971 55951 4.50527 2.44531 0.144233 0.0346665 2024-09-15T03:07:14.131 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:14.136 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:14.154 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-15T03:07:12.672411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":452827708,"num_objects":55281,"num_object_clones":0,"num_object_copies":221124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55308,"num_write_kb":442346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55335,"ondisk_log_size":55335,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1988604,"kb_used_data":756124,"kb_used_omap":14,"kb_used_meta":1232433,"kb_avail":375498756,"statfs":{"total":386547056640,"available":384510726144,"internally_reserved":0,"allocated":774270976,"data_stored":773638210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1262011419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,4,5,6,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34889728,"num_objects":4259,"num_object_clones":0,"num_object_copies":17036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4259,"num_write_kb":34072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"1.f","version":"31'3394","reported_seq":3436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.460539+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:10.460539+0000","last_peered":"2024-09-15T03:07:10.460539+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:10.460539+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3394,"log_dups_size":0,"ondisk_log_size":3394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27803648,"num_objects":3394,"num_object_clones":0,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3394,"num_write_kb":27152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3326","reported_seq":3388,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.239571+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:11.239571+0000","last_peered":"2024-09-15T03:07:11.239571+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:11.239571+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3326,"log_dups_size":0,"ondisk_log_size":3326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211378,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3539","reported_seq":3620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.781234+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:11.781234+0000","last_peered":"2024-09-15T03:07:11.781234+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:11.781234+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28965689,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":28294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1434}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3272","reported_seq":3335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.169340+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:11.169340+0000","last_peered":"2024-09-15T03:07:11.169340+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:11.169340+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26769010,"num_objects":3268,"num_object_clones":0,"num_object_copies":13072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3270,"num_write_kb":26149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3488","reported_seq":3544,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.514620+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:10.514620+0000","last_peered":"2024-09-15T03:07:10.514620+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:10.514620+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3488,"log_dups_size":0,"ondisk_log_size":3488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546674,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":27885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1378}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3542","reported_seq":3585,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.661336+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:10.661336+0000","last_peered":"2024-09-15T03:07:10.661336+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:10.661336+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29016064,"num_objects":3542,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3542,"num_write_kb":28336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3478","reported_seq":3543,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.124767+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:11.124767+0000","last_peered":"2024-09-15T03:07:11.124767+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:11.124767+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28440178,"num_objects":3472,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3476,"num_write_kb":27788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3529","reported_seq":3592,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.877569+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:10.877569+0000","last_peered":"2024-09-15T03:07:10.877569+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:10.877569+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3529,"log_dups_size":0,"ondisk_log_size":3529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28866162,"num_objects":3524,"num_object_clones":0,"num_object_copies":14096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3527,"num_write_kb":28204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3440","reported_seq":3497,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.488483+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:10.488483+0000","last_peered":"2024-09-15T03:07:10.488483+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:10.488483+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3440,"log_dups_size":0,"ondisk_log_size":3440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28110052,"num_objects":3432,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3437,"num_write_kb":27473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3491","reported_seq":3574,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.936538+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:11.936538+0000","last_peered":"2024-09-15T03:07:11.936538+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:11.936538+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28579442,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3518","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.598432+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:10.598432+0000","last_peered":"2024-09-15T03:07:10.598432+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:10.598432+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28774827,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3443","reported_seq":4439,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.641676+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:10.641676+0000","last_peered":"2024-09-15T03:07:10.641676+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:10.641676+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3443,"log_dups_size":0,"ondisk_log_size":3443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28195641,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":27535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3485","reported_seq":3561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.958281+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:11.958281+0000","last_peered":"2024-09-15T03:07:11.958281+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:11.958281+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28539705,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":27871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3413","reported_seq":3463,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:10.597167+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:10.597167+0000","last_peered":"2024-09-15T03:07:10.597167+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:10.597167+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3413,"log_dups_size":0,"ondisk_log_size":3413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27924082,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":27284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1373}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3471","reported_seq":3552,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.948819+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:11.948819+0000","last_peered":"2024-09-15T03:07:11.948819+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:11.948819+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28407410,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3469,"num_write_kb":27749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3506","reported_seq":3567,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.244570+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:11.244570+0000","last_peered":"2024-09-15T03:07:11.244570+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:11.244570+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3506,"log_dups_size":0,"ondisk_log_size":3506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28677746,"num_objects":3501,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":28020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":452827708,"num_objects":55281,"num_object_clones":0,"num_object_copies":221124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55308,"num_write_kb":442346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":774656000,"data_stored":774656000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55335,"ondisk_log_size":55335,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530876,"kb_used_data":220532,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93840964,"statfs":{"total":96636764160,"available":96093147136,"internally_reserved":0,"allocated":225824768,"data_stored":225665790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,4,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.33600000000000002},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.21199999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.318},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.36899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.39300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532028,"kb_used_data":222196,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93839812,"statfs":{"total":96636764160,"available":96091967488,"internally_reserved":0,"allocated":227528704,"data_stored":227372872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":6.6820000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.27100000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":7.6130000000000004},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":6.5919999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":9.1579999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.30499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394156,"kb_used_data":92132,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93977684,"statfs":{"total":96636764160,"available":96233148416,"internally_reserved":0,"allocated":94343168,"data_stored":94184190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":4.0019999999999998},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.23799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":3.7160000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":3.9369999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":531544,"kb_used_data":221264,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93840296,"statfs":{"total":96636764160,"available":96092463104,"internally_reserved":0,"allocated":226574336,"data_stored":226415358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.495}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":2.0489999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.27900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.24399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226496512,"data_stored":226496512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95068160,"data_stored":95068160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227409920,"data_stored":227409920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225681408,"data_stored":225681408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:14.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:07:14.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3394", 'reported_seq': 3436, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:10.460539+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:07:10.460539+0000', 'last_peered': '2024-09-15T03:07:10.460539+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:07:10.460539+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3394, 'log_dups_size': 0, 'ondisk_log_size': 3394, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27803648, 'num_objects': 3394, 'num_object_clones': 0, 'num_object_copies': 13576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3394, 'num_write_kb': 27152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1418}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:14.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:07:14.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3326", 'reported_seq': 3388, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:11.239571+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:07:11.239571+0000', 'last_peered': '2024-09-15T03:07:11.239571+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:07:11.239571+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3326, 'log_dups_size': 0, 'ondisk_log_size': 3326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27211378, 'num_objects': 3322, 'num_object_clones': 0, 'num_object_copies': 13288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3324, 'num_write_kb': 26588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1324}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:14.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:07:14.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3539", 'reported_seq': 3620, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:11.781234+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:07:11.781234+0000', 'last_peered': '2024-09-15T03:07:11.781234+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:07:11.781234+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3539, 'log_dups_size': 0, 'ondisk_log_size': 3539, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28965689, 'num_objects': 3536, 'num_object_clones': 0, 'num_object_copies': 14144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3536, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3538, 'num_write_kb': 28294, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1434}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:14.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:07:14.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3272", 'reported_seq': 3335, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:11.169340+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:07:11.169340+0000', 'last_peered': '2024-09-15T03:07:11.169340+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:07:11.169340+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3272, 'log_dups_size': 0, 'ondisk_log_size': 3272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26769010, 'num_objects': 3268, 'num_object_clones': 0, 'num_object_copies': 13072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3270, 'num_write_kb': 26149, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1325}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:14.156 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:07:14.156 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3488", 'reported_seq': 3544, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:10.514620+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:07:10.514620+0000', 'last_peered': '2024-09-15T03:07:10.514620+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:07:10.514620+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3488, 'log_dups_size': 0, 'ondisk_log_size': 3488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28546674, 'num_objects': 3485, 'num_object_clones': 0, 'num_object_copies': 13940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3486, 'num_write_kb': 27885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1378}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3542", 'reported_seq': 3585, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:10.661336+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:07:10.661336+0000', 'last_peered': '2024-09-15T03:07:10.661336+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:07:10.661336+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3542, 'log_dups_size': 0, 'ondisk_log_size': 3542, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29016064, 'num_objects': 3542, 'num_object_clones': 0, 'num_object_copies': 14168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3542, 'num_write_kb': 28336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3478", 'reported_seq': 3543, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:11.124767+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:07:11.124767+0000', 'last_peered': '2024-09-15T03:07:11.124767+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:07:11.124767+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3478, 'log_dups_size': 0, 'ondisk_log_size': 3478, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28440178, 'num_objects': 3472, 'num_object_clones': 0, 'num_object_copies': 13888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3476, 'num_write_kb': 27788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1420}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3529", 'reported_seq': 3592, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:10.877569+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:07:10.877569+0000', 'last_peered': '2024-09-15T03:07:10.877569+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:07:10.877569+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3529, 'log_dups_size': 0, 'ondisk_log_size': 3529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28866162, 'num_objects': 3524, 'num_object_clones': 0, 'num_object_copies': 14096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3527, 'num_write_kb': 28204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1443}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3440", 'reported_seq': 3497, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:10.488483+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:07:10.488483+0000', 'last_peered': '2024-09-15T03:07:10.488483+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:07:10.488483+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3440, 'log_dups_size': 0, 'ondisk_log_size': 3440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28110052, 'num_objects': 3432, 'num_object_clones': 0, 'num_object_copies': 13728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3437, 'num_write_kb': 27473, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1390}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:14.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:07:14.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3491", 'reported_seq': 3574, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:11.936538+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:07:11.936538+0000', 'last_peered': '2024-09-15T03:07:11.936538+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:07:11.936538+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3491, 'log_dups_size': 0, 'ondisk_log_size': 3491, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28579442, 'num_objects': 3489, 'num_object_clones': 0, 'num_object_copies': 13956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3489, 'num_write_kb': 27910, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1431}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:14.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:07:14.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3518", 'reported_seq': 3576, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:10.598432+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:07:10.598432+0000', 'last_peered': '2024-09-15T03:07:10.598432+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:07:10.598432+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3518, 'log_dups_size': 0, 'ondisk_log_size': 3518, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28774827, 'num_objects': 3513, 'num_object_clones': 0, 'num_object_copies': 14052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3515, 'num_write_kb': 28108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1435}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:14.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:07:14.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3443", 'reported_seq': 4439, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:10.641676+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:07:10.641676+0000', 'last_peered': '2024-09-15T03:07:10.641676+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:07:10.641676+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3443, 'log_dups_size': 0, 'ondisk_log_size': 3443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28195641, 'num_objects': 3442, 'num_object_clones': 0, 'num_object_copies': 13768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1512, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3442, 'num_write_kb': 27535, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 481, 'num_bytes_recovered': 3932160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:14.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:07:14.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3485", 'reported_seq': 3561, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:11.958281+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:07:11.958281+0000', 'last_peered': '2024-09-15T03:07:11.958281+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:07:11.958281+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3485, 'log_dups_size': 0, 'ondisk_log_size': 3485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28539705, 'num_objects': 3484, 'num_object_clones': 0, 'num_object_copies': 13936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3484, 'num_write_kb': 27871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:14.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:07:14.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3413", 'reported_seq': 3463, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:10.597167+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:07:10.597167+0000', 'last_peered': '2024-09-15T03:07:10.597167+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:07:10.597167+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3413, 'log_dups_size': 0, 'ondisk_log_size': 3413, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27924082, 'num_objects': 3409, 'num_object_clones': 0, 'num_object_copies': 13636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3411, 'num_write_kb': 27284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1373}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:14.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:07:14.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3471", 'reported_seq': 3552, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:11.948819+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:07:11.948819+0000', 'last_peered': '2024-09-15T03:07:11.948819+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:07:11.948819+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3471, 'log_dups_size': 0, 'ondisk_log_size': 3471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28407410, 'num_objects': 3468, 'num_object_clones': 0, 'num_object_copies': 13872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3469, 'num_write_kb': 27749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1480}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:14.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:07:14.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3506", 'reported_seq': 3567, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:11.244570+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:07:11.244570+0000', 'last_peered': '2024-09-15T03:07:11.244570+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:07:11.244570+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3506, 'log_dups_size': 0, 'ondisk_log_size': 3506, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28677746, 'num_objects': 3501, 'num_object_clones': 0, 'num_object_copies': 14004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3504, 'num_write_kb': 28020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:14.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:07:14.371 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 98 20 56285 56265 4.48427 2.45312 0.0418151 0.0348151 2024-09-15T03:07:15.373 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 99 20 56610 56590 4.46455 2.53906 0.0246952 0.0349756 2024-09-15T03:07:15.631 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:15.631 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32176/221124 objects degraded (14.551%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":158,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55281,"data_bytes":452827708,"bytes_used":2036330496,"bytes_avail":384510726144,"bytes_total":386547056640,"degraded_objects":32176,"degraded_total":221124,"degraded_ratio":0.14551111593495053,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":53232,"recovering_keys_per_sec":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"write_bytes_sec":2267829,"read_op_per_sec":0,"write_op_per_sec":276},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:07:15.704 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:07:15.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:16.373 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:07:16.372469+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0351335 2024-09-15T03:07:16.373 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:07:16.373 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 100 20 56924 56904 4.44444 2.45312 0.0505661 0.0351335 2024-09-15T03:07:16.829 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:16.842 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:16.872 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-15T03:07:16.673454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462559804,"num_objects":56469,"num_object_clones":0,"num_object_copies":225876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56496,"num_write_kb":451850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":551,"num_bytes_recovered":4292608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56523,"ondisk_log_size":56523,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1995456,"kb_used_data":775456,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375491904,"statfs":{"total":386547056640,"available":384503709696,"internally_reserved":0,"allocated":794066944,"data_stored":793434178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,4,17,11,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36954112,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":36088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003583"},"pg_stats":[{"pgid":"1.f","version":"31'3486","reported_seq":3528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.728329+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:15.728329+0000","last_peered":"2024-09-15T03:07:15.728329+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:15.728329+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28557312,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":27888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3426","reported_seq":3488,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.066275+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:16.066275+0000","last_peered":"2024-09-15T03:07:16.066275+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:16.066275+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28030578,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3539","reported_seq":3620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.781234+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:11.781234+0000","last_peered":"2024-09-15T03:07:11.781234+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:11.781234+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28965689,"num_objects":3536,"num_object_clones":0,"num_object_copies":14144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":28294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1434}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3390","reported_seq":3453,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.283701+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:16.283701+0000","last_peered":"2024-09-15T03:07:16.283701+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:16.283701+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27735666,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":27093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3594","reported_seq":3650,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.660231+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:15.660231+0000","last_peered":"2024-09-15T03:07:15.660231+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:15.660231+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3594,"log_dups_size":0,"ondisk_log_size":3594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29415026,"num_objects":3591,"num_object_clones":0,"num_object_copies":14364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3592,"num_write_kb":28733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1484}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3641","reported_seq":3685,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.728506+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:15.728506+0000","last_peered":"2024-09-15T03:07:15.728506+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:15.728506+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29827072,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3641,"num_write_kb":29128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3593","reported_seq":3658,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.266433+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:16.266433+0000","last_peered":"2024-09-15T03:07:16.266433+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:16.266433+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3593,"log_dups_size":0,"ondisk_log_size":3593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29382258,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3591,"num_write_kb":28708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1535}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3619","reported_seq":3682,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.232683+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:16.232683+0000","last_peered":"2024-09-15T03:07:16.232683+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:16.232683+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3619,"log_dups_size":0,"ondisk_log_size":3619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29603442,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":28924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1533}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3542","reported_seq":3599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.547323+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:15.547323+0000","last_peered":"2024-09-15T03:07:15.547323+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:15.547323+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28945636,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":28289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3491","reported_seq":3574,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.936538+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:11.936538+0000","last_peered":"2024-09-15T03:07:11.936538+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:11.936538+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28579442,"num_objects":3489,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1431}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3605","reported_seq":3663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.608330+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:15.608330+0000","last_peered":"2024-09-15T03:07:15.608330+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:15.608330+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29487531,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3602,"num_write_kb":28804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3542","reported_seq":4628,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.730834+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:15.730834+0000","last_peered":"2024-09-15T03:07:15.730834+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:15.730834+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29006649,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3541,"num_write_kb":28327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":4292608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2073}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3485","reported_seq":3561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.958281+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:11.958281+0000","last_peered":"2024-09-15T03:07:11.958281+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:11.958281+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3485,"log_dups_size":0,"ondisk_log_size":3485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28539705,"num_objects":3484,"num_object_clones":0,"num_object_copies":13936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3484,"num_write_kb":27871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3500","reported_seq":3550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.593745+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:15.593745+0000","last_peered":"2024-09-15T03:07:15.593745+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:15.593745+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636786,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3498,"num_write_kb":27980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1460}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3471","reported_seq":3552,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:11.948819+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:11.948819+0000","last_peered":"2024-09-15T03:07:11.948819+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:11.948819+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28407410,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3469,"num_write_kb":27749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1480}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3599","reported_seq":3662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.202650+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:16.202650+0000","last_peered":"2024-09-15T03:07:16.202650+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:16.202650+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29439602,"num_objects":3594,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":462559804,"num_objects":56469,"num_object_clones":0,"num_object_copies":225876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56496,"num_write_kb":451850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":551,"num_bytes_recovered":4292608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":794607616,"data_stored":794607616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56523,"ondisk_log_size":56523,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528972,"kb_used_data":226948,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93842868,"statfs":{"total":96636764160,"available":96095096832,"internally_reserved":0,"allocated":232394752,"data_stored":232235774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,6,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":17.408999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":2.1560000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":3.9820000000000002},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":1.768}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":3.9430000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":1.8}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532028,"kb_used_data":222196,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93839812,"statfs":{"total":96636764160,"available":96091967488,"internally_reserved":0,"allocated":227528704,"data_stored":227372872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":6.6820000000000004},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":0.27100000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":7.6130000000000004},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":6.5919999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":9.1579999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.30499999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409040,"kb_used_data":98696,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93962800,"statfs":{"total":96636764160,"available":96217907200,"internally_reserved":0,"allocated":101064704,"data_stored":100905726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.16900000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.20000000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.443}]}]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525416,"kb_used_data":227616,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93846424,"statfs":{"total":96636764160,"available":96098738176,"internally_reserved":0,"allocated":233078784,"data_stored":232919806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,7,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.5}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":8.1159999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":10.826000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.73699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233029632,"data_stored":233029632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101818368,"data_stored":101818368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":227409920,"data_stored":227409920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":232349696,"data_stored":232349696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:17.374 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 101 20 57233 57213 4.42429 2.41406 0.0144756 0.0353002 2024-09-15T03:07:18.374 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 102 20 57561 57541 4.40604 2.5625 0.0413211 0.0354458 2024-09-15T03:07:19.375 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 103 20 57898 57878 4.38882 2.63281 0.0836251 0.0355865 2024-09-15T03:07:19.875 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:20.374 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 104 20 58223 58203 4.37104 2.53906 0.0373715 0.035727 2024-09-15T03:07:21.170 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:21.188 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:21.216 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-15T03:07:20.675016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465926716,"num_objects":56880,"num_object_clones":0,"num_object_copies":227520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56907,"num_write_kb":455138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":551,"num_bytes_recovered":4292608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56934,"ondisk_log_size":56934,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1993380,"kb_used_data":781700,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375493980,"statfs":{"total":386547056640,"available":384505835520,"internally_reserved":0,"allocated":800460800,"data_stored":799828034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,8,8,17,9,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26656768,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3254,"num_write_kb":26032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":712704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004259"},"pg_stats":[{"pgid":"1.f","version":"31'3486","reported_seq":3528,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.728329+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:15.728329+0000","last_peered":"2024-09-15T03:07:15.728329+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:15.728329+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3486,"log_dups_size":0,"ondisk_log_size":3486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28557312,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":27888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3426","reported_seq":3488,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.066275+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:16.066275+0000","last_peered":"2024-09-15T03:07:16.066275+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:16.066275+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28030578,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3424,"num_write_kb":27388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3639","reported_seq":3720,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.913048+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:16.913048+0000","last_peered":"2024-09-15T03:07:16.913048+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:16.913048+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29784889,"num_objects":3636,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3638,"num_write_kb":29094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3390","reported_seq":3453,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.283701+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:16.283701+0000","last_peered":"2024-09-15T03:07:16.283701+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:16.283701+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27735666,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3388,"num_write_kb":27093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3594","reported_seq":3650,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.660231+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:15.660231+0000","last_peered":"2024-09-15T03:07:15.660231+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:15.660231+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3594,"log_dups_size":0,"ondisk_log_size":3594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29415026,"num_objects":3591,"num_object_clones":0,"num_object_copies":14364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3592,"num_write_kb":28733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1484}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3641","reported_seq":3685,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.728506+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:15.728506+0000","last_peered":"2024-09-15T03:07:15.728506+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:15.728506+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3641,"log_dups_size":0,"ondisk_log_size":3641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29827072,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3641,"num_write_kb":29128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3593","reported_seq":3658,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.266433+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:16.266433+0000","last_peered":"2024-09-15T03:07:16.266433+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:16.266433+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3593,"log_dups_size":0,"ondisk_log_size":3593,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29382258,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3591,"num_write_kb":28708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1535}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3619","reported_seq":3682,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.232683+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:16.232683+0000","last_peered":"2024-09-15T03:07:16.232683+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:16.232683+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3619,"log_dups_size":0,"ondisk_log_size":3619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29603442,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":28924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1533}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3542","reported_seq":3599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.547323+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:15.547323+0000","last_peered":"2024-09-15T03:07:15.547323+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:15.547323+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28945636,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":28289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3589","reported_seq":3672,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.930325+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:16.930325+0000","last_peered":"2024-09-15T03:07:16.930325+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:16.930325+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3589,"log_dups_size":0,"ondisk_log_size":3589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29382258,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3587,"num_write_kb":28694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3605","reported_seq":3663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.608330+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:15.608330+0000","last_peered":"2024-09-15T03:07:15.608330+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:15.608330+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3605,"log_dups_size":0,"ondisk_log_size":3605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29487531,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3602,"num_write_kb":28804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3542","reported_seq":4628,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.730834+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:15.730834+0000","last_peered":"2024-09-15T03:07:15.730834+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:15.730834+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29006649,"num_objects":3541,"num_object_clones":0,"num_object_copies":14164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3541,"num_write_kb":28327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":525,"num_bytes_recovered":4292608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2073}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3582","reported_seq":3659,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.966866+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:16.966866+0000","last_peered":"2024-09-15T03:07:16.966866+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:16.966866+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3582,"log_dups_size":0,"ondisk_log_size":3582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29334329,"num_objects":3581,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3581,"num_write_kb":28647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1542}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3500","reported_seq":3550,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:15.593745+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:15.593745+0000","last_peered":"2024-09-15T03:07:15.593745+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:15.593745+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3500,"log_dups_size":0,"ondisk_log_size":3500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636786,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3498,"num_write_kb":27980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1460}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3587","reported_seq":3668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.673903+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:16.673903+0000","last_peered":"2024-09-15T03:07:16.673903+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:16.673903+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3587,"log_dups_size":0,"ondisk_log_size":3587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29357682,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3585,"num_write_kb":28677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3599","reported_seq":3662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:16.202650+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:16.202650+0000","last_peered":"2024-09-15T03:07:16.202650+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:16.202650+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29439602,"num_objects":3594,"num_object_clones":0,"num_object_copies":14376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":465926716,"num_objects":56880,"num_object_clones":0,"num_object_copies":227520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56907,"num_write_kb":455138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":551,"num_bytes_recovered":4292608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":801107968,"data_stored":801107968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56934,"ondisk_log_size":56934,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528972,"kb_used_data":226948,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93842868,"statfs":{"total":96636764160,"available":96095096832,"internally_reserved":0,"allocated":232394752,"data_stored":232235774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,6,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":17.408999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":2.1560000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":3.9820000000000002},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":1.768}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":3.9430000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":1.8}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529952,"kb_used_data":228440,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93841888,"statfs":{"total":96636764160,"available":96094093312,"internally_reserved":0,"allocated":233922560,"data_stored":233766728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,0,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":23.280000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":19.172999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":19.343},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":23.311}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":21.600999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":19.274000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409040,"kb_used_data":98696,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93962800,"statfs":{"total":96636764160,"available":96217907200,"internally_reserved":0,"allocated":101064704,"data_stored":100905726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.16900000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.20000000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.443}]}]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525416,"kb_used_data":227616,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93846424,"statfs":{"total":96636764160,"available":96098738176,"internally_reserved":0,"allocated":233078784,"data_stored":232919806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,7,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.5}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":8.1159999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":10.826000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.73699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233029632,"data_stored":233029632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101818368,"data_stored":101818368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233910272,"data_stored":233910272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":232349696,"data_stored":232349696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:21.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:21.374 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 105 20 58544 58524 4.35329 2.50781 0.0346793 0.0358738 2024-09-15T03:07:22.374 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 106 20 58870 58850 4.33625 2.54688 0.0668615 0.0360186 2024-09-15T03:07:22.788 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:22.796 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:22.818 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-09-15T03:07:22.676407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":479353404,"num_objects":58519,"num_object_clones":0,"num_object_copies":234076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58546,"num_write_kb":468250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":4562944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58573,"ondisk_log_size":58573,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2023752,"kb_used_data":807912,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375463608,"statfs":{"total":386547056640,"available":384474734592,"internally_reserved":0,"allocated":827301888,"data_stored":826669122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,7,8,5,15,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40083456,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4893,"num_write_kb":39144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005194"},"pg_stats":[{"pgid":"1.f","version":"31'3602","reported_seq":3643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.761767+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:20.761767+0000","last_peered":"2024-09-15T03:07:20.761767+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:20.761767+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29507584,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3602,"num_write_kb":28816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3514","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.063923+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:21.063923+0000","last_peered":"2024-09-15T03:07:21.063923+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:21.063923+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28751474,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":28092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3749","reported_seq":3830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.852902+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:21.852902+0000","last_peered":"2024-09-15T03:07:21.852902+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:21.852902+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3749,"log_dups_size":0,"ondisk_log_size":3749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30686009,"num_objects":3746,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3748,"num_write_kb":29974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1644}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3489","reported_seq":3552,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.140336+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:21.140336+0000","last_peered":"2024-09-15T03:07:21.140336+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:21.140336+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3489,"log_dups_size":0,"ondisk_log_size":3489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546674,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3487,"num_write_kb":27885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3693","reported_seq":3749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.719465+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:20.719465+0000","last_peered":"2024-09-15T03:07:20.719465+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:20.719465+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3693,"log_dups_size":0,"ondisk_log_size":3693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30226034,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":29525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3758","reported_seq":3802,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.753168+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:20.753168+0000","last_peered":"2024-09-15T03:07:20.753168+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:20.753168+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3758,"log_dups_size":0,"ondisk_log_size":3758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30785536,"num_objects":3758,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3758,"num_write_kb":30064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1702}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3696","reported_seq":3761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.233665+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:21.233665+0000","last_peered":"2024-09-15T03:07:21.233665+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:21.233665+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3696,"log_dups_size":0,"ondisk_log_size":3696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30226034,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3694,"num_write_kb":29532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1638}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3725","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.200557+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:21.200557+0000","last_peered":"2024-09-15T03:07:21.200557+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:21.200557+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3725,"log_dups_size":0,"ondisk_log_size":3725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471794,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":29772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3636","reported_seq":3693,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.534043+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:20.534043+0000","last_peered":"2024-09-15T03:07:20.534043+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:20.534043+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3636,"log_dups_size":0,"ondisk_log_size":3636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29715684,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3633,"num_write_kb":29041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3694","reported_seq":3777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.981387+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:21.981387+0000","last_peered":"2024-09-15T03:07:21.981387+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:21.981387+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30242418,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3692,"num_write_kb":29534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3715","reported_seq":3773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.653471+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:20.653471+0000","last_peered":"2024-09-15T03:07:20.653471+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:20.653471+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3715,"log_dups_size":0,"ondisk_log_size":3715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30388651,"num_objects":3710,"num_object_clones":0,"num_object_copies":14840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3712,"num_write_kb":29684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3642","reported_seq":4794,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.779725+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:20.779725+0000","last_peered":"2024-09-15T03:07:20.779725+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:20.779725+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29825849,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3641,"num_write_kb":29127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":558,"num_bytes_recovered":4562944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2206}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3676","reported_seq":3753,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.770714+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:21.770714+0000","last_peered":"2024-09-15T03:07:21.770714+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:21.770714+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30104377,"num_objects":3675,"num_object_clones":0,"num_object_copies":14700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3675,"num_write_kb":29399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3599","reported_seq":3649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.769535+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:20.769535+0000","last_peered":"2024-09-15T03:07:20.769535+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:20.769535+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447794,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3686","reported_seq":3767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.994382+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:21.994382+0000","last_peered":"2024-09-15T03:07:21.994382+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:21.994382+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3686,"log_dups_size":0,"ondisk_log_size":3686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30168690,"num_objects":3683,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":29469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1695}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3699","reported_seq":3761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.208029+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:21.208029+0000","last_peered":"2024-09-15T03:07:21.208029+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:21.208029+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30258802,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3697,"num_write_kb":29564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":479353404,"num_objects":58519,"num_object_clones":0,"num_object_copies":234076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58546,"num_write_kb":468250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":4562944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":828137472,"data_stored":828137472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58573,"ondisk_log_size":58573,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539684,"kb_used_data":233500,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93832156,"statfs":{"total":96636764160,"available":96084127744,"internally_reserved":0,"allocated":239104000,"data_stored":238945022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,2,1,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":1.1399999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.379},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.32700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540700,"kb_used_data":235028,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93831140,"statfs":{"total":96636764160,"available":96083087360,"internally_reserved":0,"allocated":240668672,"data_stored":240512840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.1949999999999998},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.24299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":8.8300000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":5.077},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.40500000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403052,"kb_used_data":105188,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93968788,"statfs":{"total":96636764160,"available":96224038912,"internally_reserved":0,"allocated":107712512,"data_stored":107553534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.182},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.31}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.34399999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":2.5150000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540316,"kb_used_data":234196,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93831524,"statfs":{"total":96636764160,"available":96083480576,"internally_reserved":0,"allocated":239816704,"data_stored":239657726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.438},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.29999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.49399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239730688,"data_stored":239730688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":108699648,"data_stored":108699648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240660480,"data_stored":240660480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":239046656,"data_stored":239046656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:22.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:07:22.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3602", 'reported_seq': 3643, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:20.761767+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:07:20.761767+0000', 'last_peered': '2024-09-15T03:07:20.761767+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:07:20.761767+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3602, 'log_dups_size': 0, 'ondisk_log_size': 3602, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29507584, 'num_objects': 3602, 'num_object_clones': 0, 'num_object_copies': 14408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3602, 'num_write_kb': 28816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1626}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:22.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3514", 'reported_seq': 3576, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.063923+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:07:21.063923+0000', 'last_peered': '2024-09-15T03:07:21.063923+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:07:21.063923+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3514, 'log_dups_size': 0, 'ondisk_log_size': 3514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28751474, 'num_objects': 3510, 'num_object_clones': 0, 'num_object_copies': 14040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3512, 'num_write_kb': 28092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3749", 'reported_seq': 3830, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.852902+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:07:21.852902+0000', 'last_peered': '2024-09-15T03:07:21.852902+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:07:21.852902+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3749, 'log_dups_size': 0, 'ondisk_log_size': 3749, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30686009, 'num_objects': 3746, 'num_object_clones': 0, 'num_object_copies': 14984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3748, 'num_write_kb': 29974, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1644}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3489", 'reported_seq': 3552, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.140336+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:07:21.140336+0000', 'last_peered': '2024-09-15T03:07:21.140336+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:07:21.140336+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3489, 'log_dups_size': 0, 'ondisk_log_size': 3489, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28546674, 'num_objects': 3485, 'num_object_clones': 0, 'num_object_copies': 13940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3487, 'num_write_kb': 27885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1542}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3693", 'reported_seq': 3749, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:20.719465+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:07:20.719465+0000', 'last_peered': '2024-09-15T03:07:20.719465+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:07:20.719465+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3693, 'log_dups_size': 0, 'ondisk_log_size': 3693, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30226034, 'num_objects': 3690, 'num_object_clones': 0, 'num_object_copies': 14760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3691, 'num_write_kb': 29525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1583}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3758", 'reported_seq': 3802, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:20.753168+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:07:20.753168+0000', 'last_peered': '2024-09-15T03:07:20.753168+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:07:20.753168+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3758, 'log_dups_size': 0, 'ondisk_log_size': 3758, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30785536, 'num_objects': 3758, 'num_object_clones': 0, 'num_object_copies': 15032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3758, 'num_write_kb': 30064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1702}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:22.820 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3696", 'reported_seq': 3761, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.233665+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:07:21.233665+0000', 'last_peered': '2024-09-15T03:07:21.233665+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:07:21.233665+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3696, 'log_dups_size': 0, 'ondisk_log_size': 3696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30226034, 'num_objects': 3690, 'num_object_clones': 0, 'num_object_copies': 14760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3694, 'num_write_kb': 29532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1638}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3725", 'reported_seq': 3788, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.200557+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:07:21.200557+0000', 'last_peered': '2024-09-15T03:07:21.200557+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:07:21.200557+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3725, 'log_dups_size': 0, 'ondisk_log_size': 3725, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30471794, 'num_objects': 3720, 'num_object_clones': 0, 'num_object_copies': 14880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3723, 'num_write_kb': 29772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3636", 'reported_seq': 3693, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:20.534043+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:07:20.534043+0000', 'last_peered': '2024-09-15T03:07:20.534043+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:07:20.534043+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3636, 'log_dups_size': 0, 'ondisk_log_size': 3636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29715684, 'num_objects': 3628, 'num_object_clones': 0, 'num_object_copies': 14512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3633, 'num_write_kb': 29041, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1586}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3694", 'reported_seq': 3777, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.981387+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:07:21.981387+0000', 'last_peered': '2024-09-15T03:07:21.981387+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:07:21.981387+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3694, 'log_dups_size': 0, 'ondisk_log_size': 3694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30242418, 'num_objects': 3692, 'num_object_clones': 0, 'num_object_copies': 14768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3692, 'num_write_kb': 29534, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1634}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3715", 'reported_seq': 3773, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:20.653471+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:07:20.653471+0000', 'last_peered': '2024-09-15T03:07:20.653471+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:07:20.653471+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3715, 'log_dups_size': 0, 'ondisk_log_size': 3715, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30388651, 'num_objects': 3710, 'num_object_clones': 0, 'num_object_copies': 14840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3712, 'num_write_kb': 29684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1632}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:07:22.821 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3642", 'reported_seq': 4794, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:20.779725+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:07:20.779725+0000', 'last_peered': '2024-09-15T03:07:20.779725+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:07:20.779725+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3642, 'log_dups_size': 0, 'ondisk_log_size': 3642, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29825849, 'num_objects': 3641, 'num_object_clones': 0, 'num_object_copies': 14564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1435, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3641, 'num_write_kb': 29127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 558, 'num_bytes_recovered': 4562944, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2206}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:22.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:07:22.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3676", 'reported_seq': 3753, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.770714+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:07:21.770714+0000', 'last_peered': '2024-09-15T03:07:21.770714+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:07:21.770714+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3676, 'log_dups_size': 0, 'ondisk_log_size': 3676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30104377, 'num_objects': 3675, 'num_object_clones': 0, 'num_object_copies': 14700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3675, 'num_write_kb': 29399, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1636}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:22.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:07:22.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3599", 'reported_seq': 3649, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:20.769535+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:07:20.769535+0000', 'last_peered': '2024-09-15T03:07:20.769535+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:07:20.769535+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3599, 'log_dups_size': 0, 'ondisk_log_size': 3599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29447794, 'num_objects': 3595, 'num_object_clones': 0, 'num_object_copies': 14380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3597, 'num_write_kb': 28772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:22.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:07:22.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3686", 'reported_seq': 3767, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.994382+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:07:21.994382+0000', 'last_peered': '2024-09-15T03:07:21.994382+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:07:21.994382+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3686, 'log_dups_size': 0, 'ondisk_log_size': 3686, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30168690, 'num_objects': 3683, 'num_object_clones': 0, 'num_object_copies': 14732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3684, 'num_write_kb': 29469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1695}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:22.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:07:22.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3699", 'reported_seq': 3761, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:21.208029+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:07:21.208029+0000', 'last_peered': '2024-09-15T03:07:21.208029+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:07:21.208029+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3699, 'log_dups_size': 0, 'ondisk_log_size': 3699, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30258802, 'num_objects': 3694, 'num_object_clones': 0, 'num_object_copies': 14776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3697, 'num_write_kb': 29564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1612}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:22.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:07:23.374 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 107 20 59174 59154 4.31792 2.375 0.143787 0.0361661 2024-09-15T03:07:24.375 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 108 20 59496 59476 4.30122 2.51562 0.0421332 0.0363135 2024-09-15T03:07:24.384 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:24.384 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32099/234076 objects degraded (13.713%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":167,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58519,"data_bytes":479353404,"bytes_used":2072322048,"bytes_avail":384474734592,"bytes_total":386547056640,"degraded_objects":32099,"degraded_total":234076,"degraded_ratio":0.13713067550710026,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":81884,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"write_bytes_sec":3338842,"read_op_per_sec":0,"write_op_per_sec":407},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:07:24.446 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:07:24.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:25.375 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 109 20 59803 59783 4.28376 2.39844 0.0876559 0.036454 2024-09-15T03:07:25.821 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:25.834 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:25.860 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-15T03:07:24.676973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":479353404,"num_objects":58519,"num_object_clones":0,"num_object_copies":234076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58546,"num_write_kb":468250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":4562944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58573,"ondisk_log_size":58573,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2023752,"kb_used_data":807912,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375463608,"statfs":{"total":386547056640,"available":384474734592,"internally_reserved":0,"allocated":827301888,"data_stored":826669122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,7,8,5,15,14,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26525696,"num_objects":3238,"num_object_clones":0,"num_object_copies":12952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3238,"num_write_kb":25904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004561"},"pg_stats":[{"pgid":"1.f","version":"31'3602","reported_seq":3643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.761767+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:20.761767+0000","last_peered":"2024-09-15T03:07:20.761767+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:20.761767+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3602,"log_dups_size":0,"ondisk_log_size":3602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29507584,"num_objects":3602,"num_object_clones":0,"num_object_copies":14408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3602,"num_write_kb":28816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3514","reported_seq":3576,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.063923+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:21.063923+0000","last_peered":"2024-09-15T03:07:21.063923+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:21.063923+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3514,"log_dups_size":0,"ondisk_log_size":3514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28751474,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":28092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1512}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3749","reported_seq":3830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.852902+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:21.852902+0000","last_peered":"2024-09-15T03:07:21.852902+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:21.852902+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3749,"log_dups_size":0,"ondisk_log_size":3749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30686009,"num_objects":3746,"num_object_clones":0,"num_object_copies":14984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3748,"num_write_kb":29974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1644}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3489","reported_seq":3552,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.140336+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:21.140336+0000","last_peered":"2024-09-15T03:07:21.140336+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:21.140336+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3489,"log_dups_size":0,"ondisk_log_size":3489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546674,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3487,"num_write_kb":27885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1542}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3693","reported_seq":3749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.719465+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:20.719465+0000","last_peered":"2024-09-15T03:07:20.719465+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:20.719465+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3693,"log_dups_size":0,"ondisk_log_size":3693,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30226034,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3691,"num_write_kb":29525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3758","reported_seq":3802,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.753168+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:20.753168+0000","last_peered":"2024-09-15T03:07:20.753168+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:20.753168+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3758,"log_dups_size":0,"ondisk_log_size":3758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30785536,"num_objects":3758,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3758,"num_write_kb":30064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1702}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3696","reported_seq":3761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.233665+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:21.233665+0000","last_peered":"2024-09-15T03:07:21.233665+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:21.233665+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3696,"log_dups_size":0,"ondisk_log_size":3696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30226034,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3694,"num_write_kb":29532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1638}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3725","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.200557+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:21.200557+0000","last_peered":"2024-09-15T03:07:21.200557+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:21.200557+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3725,"log_dups_size":0,"ondisk_log_size":3725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30471794,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":29772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3636","reported_seq":3693,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.534043+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:20.534043+0000","last_peered":"2024-09-15T03:07:20.534043+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:20.534043+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3636,"log_dups_size":0,"ondisk_log_size":3636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29715684,"num_objects":3628,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3633,"num_write_kb":29041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3694","reported_seq":3777,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.981387+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:21.981387+0000","last_peered":"2024-09-15T03:07:21.981387+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:21.981387+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30242418,"num_objects":3692,"num_object_clones":0,"num_object_copies":14768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3692,"num_write_kb":29534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3715","reported_seq":3773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.653471+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:20.653471+0000","last_peered":"2024-09-15T03:07:20.653471+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:20.653471+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3715,"log_dups_size":0,"ondisk_log_size":3715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30388651,"num_objects":3710,"num_object_clones":0,"num_object_copies":14840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3712,"num_write_kb":29684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3642","reported_seq":4794,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.779725+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:20.779725+0000","last_peered":"2024-09-15T03:07:20.779725+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:20.779725+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29825849,"num_objects":3641,"num_object_clones":0,"num_object_copies":14564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3641,"num_write_kb":29127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":558,"num_bytes_recovered":4562944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2206}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3676","reported_seq":3753,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.770714+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:21.770714+0000","last_peered":"2024-09-15T03:07:21.770714+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:21.770714+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30104377,"num_objects":3675,"num_object_clones":0,"num_object_copies":14700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3675,"num_write_kb":29399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1636}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3599","reported_seq":3649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:20.769535+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:20.769535+0000","last_peered":"2024-09-15T03:07:20.769535+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:20.769535+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29447794,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3686","reported_seq":3767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.994382+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:21.994382+0000","last_peered":"2024-09-15T03:07:21.994382+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:21.994382+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3686,"log_dups_size":0,"ondisk_log_size":3686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30168690,"num_objects":3683,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":29469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1695}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3699","reported_seq":3761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:21.208029+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:21.208029+0000","last_peered":"2024-09-15T03:07:21.208029+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:21.208029+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30258802,"num_objects":3694,"num_object_clones":0,"num_object_copies":14776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3697,"num_write_kb":29564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1612}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":479353404,"num_objects":58519,"num_object_clones":0,"num_object_copies":234076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58546,"num_write_kb":468250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":4562944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":828137472,"data_stored":828137472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58573,"ondisk_log_size":58573,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539684,"kb_used_data":233500,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93832156,"statfs":{"total":96636764160,"available":96084127744,"internally_reserved":0,"allocated":239104000,"data_stored":238945022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,2,1,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":1.1399999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.35199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.379},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.32700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540700,"kb_used_data":235028,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93831140,"statfs":{"total":96636764160,"available":96083087360,"internally_reserved":0,"allocated":240668672,"data_stored":240512840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.1949999999999998},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.24299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":8.8300000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":5.077},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":0.40500000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403052,"kb_used_data":105188,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93968788,"statfs":{"total":96636764160,"available":96224038912,"internally_reserved":0,"allocated":107712512,"data_stored":107553534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.182},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.31}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.34399999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":2.5150000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540316,"kb_used_data":234196,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93831524,"statfs":{"total":96636764160,"available":96083480576,"internally_reserved":0,"allocated":239816704,"data_stored":239657726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.438},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.29999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:06:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":6.8659999999999997,"5min":6.8659999999999997,"15min":6.8659999999999997},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.6269999999999998,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.49399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239730688,"data_stored":239730688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":108699648,"data_stored":108699648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240660480,"data_stored":240660480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":239046656,"data_stored":239046656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:26.375 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 110 20 60123 60103 4.26755 2.5 0.0887613 0.0365929 2024-09-15T03:07:27.375 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 111 20 60453 60433 4.25233 2.57812 0.102102 0.0367285 2024-09-15T03:07:28.375 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 112 20 60770 60750 4.23647 2.47656 0.0868118 0.0368663 2024-09-15T03:07:28.863 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:29.375 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 113 20 61087 61067 4.2209 2.47656 0.0518169 0.0370031 2024-09-15T03:07:30.341 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:30.352 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:30.375 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 114 20 61378 61358 4.20382 2.27344 0.0497153 0.0371529 2024-09-15T03:07:30.405 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-15T03:07:28.678016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":492419644,"num_objects":60114,"num_object_clones":0,"num_object_copies":240456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60141,"num_write_kb":481010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":4980736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60168,"ondisk_log_size":60168,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2058960,"kb_used_data":834800,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375428400,"statfs":{"total":386547056640,"available":384438681600,"internally_reserved":0,"allocated":854835200,"data_stored":854202434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,11,11,8,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29859840,"num_objects":3645,"num_object_clones":0,"num_object_copies":14580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3645,"num_write_kb":29160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":688128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004562"},"pg_stats":[{"pgid":"1.f","version":"31'3708","reported_seq":3750,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.787496+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:25.787496+0000","last_peered":"2024-09-15T03:07:25.787496+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:25.787496+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3708,"log_dups_size":0,"ondisk_log_size":3708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30375936,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3708,"num_write_kb":29664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3603","reported_seq":3665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.107200+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:26.107200+0000","last_peered":"2024-09-15T03:07:26.107200+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:26.107200+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29480562,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3601,"num_write_kb":28804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3845","reported_seq":3926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.927416+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:26.927416+0000","last_peered":"2024-09-15T03:07:26.927416+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:26.927416+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31472441,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":30742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1740}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3613","reported_seq":3676,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.285835+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:26.285835+0000","last_peered":"2024-09-15T03:07:26.285835+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:26.285835+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562482,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":28877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3787","reported_seq":3843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.751722+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:25.751722+0000","last_peered":"2024-09-15T03:07:25.751722+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:25.751722+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30996082,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3785,"num_write_kb":30277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3851","reported_seq":3895,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.802856+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:25.802856+0000","last_peered":"2024-09-15T03:07:25.802856+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:25.802856+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31547392,"num_objects":3851,"num_object_clones":0,"num_object_copies":15404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":30808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3796","reported_seq":3860,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.308581+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:26.308581+0000","last_peered":"2024-09-15T03:07:26.308581+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:26.308581+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31045234,"num_objects":3790,"num_object_clones":0,"num_object_copies":15160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3794,"num_write_kb":30332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1738}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3837","reported_seq":3900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.254956+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:26.254956+0000","last_peered":"2024-09-15T03:07:26.254956+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:26.254956+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31389298,"num_objects":3832,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":30668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3741","reported_seq":3797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.851949+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:25.851949+0000","last_peered":"2024-09-15T03:07:25.851949+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:25.851949+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3741,"log_dups_size":0,"ondisk_log_size":3741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30575844,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3738,"num_write_kb":29881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3790","reported_seq":3873,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:27.002546+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:27.002546+0000","last_peered":"2024-09-15T03:07:27.002546+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:27.002546+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31028850,"num_objects":3788,"num_object_clones":0,"num_object_copies":15152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":30302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3809","reported_seq":3867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.690282+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:25.690282+0000","last_peered":"2024-09-15T03:07:25.690282+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:25.690282+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31158699,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3806,"num_write_kb":30436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3725","reported_seq":4979,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.856851+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:25.856851+0000","last_peered":"2024-09-15T03:07:25.856851+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:25.856851+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3725,"log_dups_size":0,"ondisk_log_size":3725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30505785,"num_objects":3724,"num_object_clones":0,"num_object_copies":14896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":29791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":4980736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3778","reported_seq":3854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:27.061079+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:27.061079+0000","last_peered":"2024-09-15T03:07:27.061079+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:27.061079+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30939961,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3777,"num_write_kb":30215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1738}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3701","reported_seq":3751,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.776381+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:25.776381+0000","last_peered":"2024-09-15T03:07:25.776381+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:25.776381+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30283378,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":29588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1661}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3786","reported_seq":3867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.942444+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:26.942444+0000","last_peered":"2024-09-15T03:07:26.942444+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:26.942444+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30987890,"num_objects":3783,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3798","reported_seq":3861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.295530+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:26.295530+0000","last_peered":"2024-09-15T03:07:26.295530+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:26.295530+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31069810,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3796,"num_write_kb":30356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1711}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":492419644,"num_objects":60114,"num_object_clones":0,"num_object_copies":240456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60141,"num_write_kb":481010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":4980736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":854470656,"data_stored":854470656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60168,"ondisk_log_size":60168,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":537780,"kb_used_data":239916,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93834060,"statfs":{"total":96636764160,"available":96086077440,"internally_reserved":0,"allocated":245673984,"data_stored":245515006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":16.027999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.55800000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":5.7409999999999997},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":5.7699999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.51900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551092,"kb_used_data":241260,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93820748,"statfs":{"total":96636764160,"available":96072445952,"internally_reserved":0,"allocated":247050240,"data_stored":246894408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":6.9109999999999996},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.994}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":2.0270000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":2.306}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":9.6980000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":2.056}]}]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419152,"kb_used_data":112968,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93952688,"statfs":{"total":96636764160,"available":96207552512,"internally_reserved":0,"allocated":115679232,"data_stored":115520254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":2.3220000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.57099999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.22600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":1.302}]}]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550936,"kb_used_data":240656,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93820904,"statfs":{"total":96636764160,"available":96072605696,"internally_reserved":0,"allocated":246431744,"data_stored":246272766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.34200000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":6.0199999999999996},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":5.7590000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246190080,"data_stored":246190080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115499008,"data_stored":115499008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247197696,"data_stored":247197696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245583872,"data_stored":245583872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:30.406 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:31.376 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 115 20 61695 61675 4.1888 2.47656 0.0166705 0.0372907 2024-09-15T03:07:31.878 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:31.886 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:31.911 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-15T03:07:30.678551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":492419644,"num_objects":60114,"num_object_clones":0,"num_object_copies":240456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60141,"num_write_kb":481010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":4980736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60168,"ondisk_log_size":60168,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2058960,"kb_used_data":834800,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375428400,"statfs":{"total":386547056640,"available":384438681600,"internally_reserved":0,"allocated":854835200,"data_stored":854202434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,11,11,8,9,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26492928,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":25872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":688128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004041"},"pg_stats":[{"pgid":"1.f","version":"31'3708","reported_seq":3750,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.787496+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:25.787496+0000","last_peered":"2024-09-15T03:07:25.787496+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:25.787496+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3708,"log_dups_size":0,"ondisk_log_size":3708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30375936,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3708,"num_write_kb":29664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3603","reported_seq":3665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.107200+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:26.107200+0000","last_peered":"2024-09-15T03:07:26.107200+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:26.107200+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29480562,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3601,"num_write_kb":28804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3845","reported_seq":3926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.927416+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:26.927416+0000","last_peered":"2024-09-15T03:07:26.927416+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:26.927416+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3845,"log_dups_size":0,"ondisk_log_size":3845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31472441,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3844,"num_write_kb":30742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1740}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3613","reported_seq":3676,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.285835+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:26.285835+0000","last_peered":"2024-09-15T03:07:26.285835+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:26.285835+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3613,"log_dups_size":0,"ondisk_log_size":3613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29562482,"num_objects":3609,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":28877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3787","reported_seq":3843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.751722+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:25.751722+0000","last_peered":"2024-09-15T03:07:25.751722+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:25.751722+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30996082,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3785,"num_write_kb":30277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3851","reported_seq":3895,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.802856+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:25.802856+0000","last_peered":"2024-09-15T03:07:25.802856+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:25.802856+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3851,"log_dups_size":0,"ondisk_log_size":3851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31547392,"num_objects":3851,"num_object_clones":0,"num_object_copies":15404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":30808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3796","reported_seq":3860,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.308581+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:26.308581+0000","last_peered":"2024-09-15T03:07:26.308581+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:26.308581+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31045234,"num_objects":3790,"num_object_clones":0,"num_object_copies":15160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3794,"num_write_kb":30332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1738}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3837","reported_seq":3900,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.254956+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:26.254956+0000","last_peered":"2024-09-15T03:07:26.254956+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:26.254956+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3837,"log_dups_size":0,"ondisk_log_size":3837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31389298,"num_objects":3832,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":30668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3741","reported_seq":3797,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.851949+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:25.851949+0000","last_peered":"2024-09-15T03:07:25.851949+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:25.851949+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3741,"log_dups_size":0,"ondisk_log_size":3741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30575844,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3738,"num_write_kb":29881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3790","reported_seq":3873,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:27.002546+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:27.002546+0000","last_peered":"2024-09-15T03:07:27.002546+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:27.002546+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31028850,"num_objects":3788,"num_object_clones":0,"num_object_copies":15152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":30302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1730}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3809","reported_seq":3867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.690282+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:25.690282+0000","last_peered":"2024-09-15T03:07:25.690282+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:25.690282+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31158699,"num_objects":3804,"num_object_clones":0,"num_object_copies":15216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3806,"num_write_kb":30436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3725","reported_seq":4979,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.856851+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:25.856851+0000","last_peered":"2024-09-15T03:07:25.856851+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:25.856851+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3725,"log_dups_size":0,"ondisk_log_size":3725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30505785,"num_objects":3724,"num_object_clones":0,"num_object_copies":14896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1384,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3724,"num_write_kb":29791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":609,"num_bytes_recovered":4980736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3778","reported_seq":3854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:27.061079+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:27.061079+0000","last_peered":"2024-09-15T03:07:27.061079+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:27.061079+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3778,"log_dups_size":0,"ondisk_log_size":3778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30939961,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3777,"num_write_kb":30215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1738}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3701","reported_seq":3751,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:25.776381+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:25.776381+0000","last_peered":"2024-09-15T03:07:25.776381+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:25.776381+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30283378,"num_objects":3697,"num_object_clones":0,"num_object_copies":14788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":29588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1661}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3786","reported_seq":3867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.942444+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:26.942444+0000","last_peered":"2024-09-15T03:07:26.942444+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:26.942444+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3786,"log_dups_size":0,"ondisk_log_size":3786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30987890,"num_objects":3783,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3784,"num_write_kb":30269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3798","reported_seq":3861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:26.295530+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:26.295530+0000","last_peered":"2024-09-15T03:07:26.295530+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:26.295530+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31069810,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3796,"num_write_kb":30356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1711}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":492419644,"num_objects":60114,"num_object_clones":0,"num_object_copies":240456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60141,"num_write_kb":481010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":4980736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":854470656,"data_stored":854470656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60168,"ondisk_log_size":60168,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":537780,"kb_used_data":239916,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93834060,"statfs":{"total":96636764160,"available":96086077440,"internally_reserved":0,"allocated":245673984,"data_stored":245515006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":16.027999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.55800000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":5.7409999999999997},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":5.7699999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.51900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551092,"kb_used_data":241260,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93820748,"statfs":{"total":96636764160,"available":96072445952,"internally_reserved":0,"allocated":247050240,"data_stored":246894408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":6.9109999999999996},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.994}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":2.0270000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":2.306}]},{"osd":3,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.5960000000000001,"5min":2.5960000000000001,"15min":2.5960000000000001},"last":9.6980000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5369999999999999,"5min":3.5369999999999999,"15min":3.5369999999999999},"last":2.056}]}]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419152,"kb_used_data":112968,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93952688,"statfs":{"total":96636764160,"available":96207552512,"internally_reserved":0,"allocated":115679232,"data_stored":115520254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":2.3220000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.57099999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.22600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":1.302}]}]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550936,"kb_used_data":240656,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93820904,"statfs":{"total":96636764160,"available":96072605696,"internally_reserved":0,"allocated":246431744,"data_stored":246272766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,5,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.34200000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":6.0199999999999996},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":5.7590000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:06:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.657,"5min":3.657,"15min":3.657},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.163,"5min":1.163,"15min":1.163},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":246190080,"data_stored":246190080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115499008,"data_stored":115499008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247197696,"data_stored":247197696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245583872,"data_stored":245583872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:31.912 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3708", 'reported_seq': 3750, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:25.787496+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:07:25.787496+0000', 'last_peered': '2024-09-15T03:07:25.787496+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:07:25.787496+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3708, 'log_dups_size': 0, 'ondisk_log_size': 3708, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30375936, 'num_objects': 3708, 'num_object_clones': 0, 'num_object_copies': 14832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3708, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3708, 'num_write_kb': 29664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1732}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3603", 'reported_seq': 3665, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:26.107200+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:07:26.107200+0000', 'last_peered': '2024-09-15T03:07:26.107200+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:07:26.107200+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3603, 'log_dups_size': 0, 'ondisk_log_size': 3603, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29480562, 'num_objects': 3599, 'num_object_clones': 0, 'num_object_copies': 14396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3601, 'num_write_kb': 28804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3845", 'reported_seq': 3926, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:26.927416+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:07:26.927416+0000', 'last_peered': '2024-09-15T03:07:26.927416+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:07:26.927416+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3845, 'log_dups_size': 0, 'ondisk_log_size': 3845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31472441, 'num_objects': 3842, 'num_object_clones': 0, 'num_object_copies': 15368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3844, 'num_write_kb': 30742, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1740}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3613", 'reported_seq': 3676, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:26.285835+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:07:26.285835+0000', 'last_peered': '2024-09-15T03:07:26.285835+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:07:26.285835+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3613, 'log_dups_size': 0, 'ondisk_log_size': 3613, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29562482, 'num_objects': 3609, 'num_object_clones': 0, 'num_object_copies': 14436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3611, 'num_write_kb': 28877, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1666}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:07:31.913 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3787", 'reported_seq': 3843, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:25.751722+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:07:25.751722+0000', 'last_peered': '2024-09-15T03:07:25.751722+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:07:25.751722+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3787, 'log_dups_size': 0, 'ondisk_log_size': 3787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30996082, 'num_objects': 3784, 'num_object_clones': 0, 'num_object_copies': 15136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3785, 'num_write_kb': 30277, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1677}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3851", 'reported_seq': 3895, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:25.802856+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:07:25.802856+0000', 'last_peered': '2024-09-15T03:07:25.802856+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:07:25.802856+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3851, 'log_dups_size': 0, 'ondisk_log_size': 3851, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31547392, 'num_objects': 3851, 'num_object_clones': 0, 'num_object_copies': 15404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3851, 'num_write_kb': 30808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1795}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3796", 'reported_seq': 3860, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:26.308581+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:07:26.308581+0000', 'last_peered': '2024-09-15T03:07:26.308581+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:07:26.308581+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3796, 'log_dups_size': 0, 'ondisk_log_size': 3796, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31045234, 'num_objects': 3790, 'num_object_clones': 0, 'num_object_copies': 15160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3794, 'num_write_kb': 30332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1738}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3837", 'reported_seq': 3900, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:26.254956+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:07:26.254956+0000', 'last_peered': '2024-09-15T03:07:26.254956+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:07:26.254956+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3837, 'log_dups_size': 0, 'ondisk_log_size': 3837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31389298, 'num_objects': 3832, 'num_object_clones': 0, 'num_object_copies': 15328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3835, 'num_write_kb': 30668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1751}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3741", 'reported_seq': 3797, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:25.851949+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:07:25.851949+0000', 'last_peered': '2024-09-15T03:07:25.851949+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:07:25.851949+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3741, 'log_dups_size': 0, 'ondisk_log_size': 3741, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30575844, 'num_objects': 3733, 'num_object_clones': 0, 'num_object_copies': 14932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3738, 'num_write_kb': 29881, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1691}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3790", 'reported_seq': 3873, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:27.002546+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:07:27.002546+0000', 'last_peered': '2024-09-15T03:07:27.002546+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:07:27.002546+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3790, 'log_dups_size': 0, 'ondisk_log_size': 3790, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31028850, 'num_objects': 3788, 'num_object_clones': 0, 'num_object_copies': 15152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3788, 'num_write_kb': 30302, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1730}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:31.914 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3809", 'reported_seq': 3867, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:25.690282+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:07:25.690282+0000', 'last_peered': '2024-09-15T03:07:25.690282+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:07:25.690282+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3809, 'log_dups_size': 0, 'ondisk_log_size': 3809, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31158699, 'num_objects': 3804, 'num_object_clones': 0, 'num_object_copies': 15216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3806, 'num_write_kb': 30436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1726}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3725", 'reported_seq': 4979, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:25.856851+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:07:25.856851+0000', 'last_peered': '2024-09-15T03:07:25.856851+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:07:25.856851+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3725, 'log_dups_size': 0, 'ondisk_log_size': 3725, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30505785, 'num_objects': 3724, 'num_object_clones': 0, 'num_object_copies': 14896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1384, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3724, 'num_write_kb': 29791, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 609, 'num_bytes_recovered': 4980736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3778", 'reported_seq': 3854, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:27.061079+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:07:27.061079+0000', 'last_peered': '2024-09-15T03:07:27.061079+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:07:27.061079+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3778, 'log_dups_size': 0, 'ondisk_log_size': 3778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30939961, 'num_objects': 3777, 'num_object_clones': 0, 'num_object_copies': 15108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3777, 'num_write_kb': 30215, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3701", 'reported_seq': 3751, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:25.776381+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:07:25.776381+0000', 'last_peered': '2024-09-15T03:07:25.776381+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:07:25.776381+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3701, 'log_dups_size': 0, 'ondisk_log_size': 3701, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30283378, 'num_objects': 3697, 'num_object_clones': 0, 'num_object_copies': 14788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3699, 'num_write_kb': 29588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3786", 'reported_seq': 3867, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:26.942444+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:07:26.942444+0000', 'last_peered': '2024-09-15T03:07:26.942444+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:07:26.942444+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3786, 'log_dups_size': 0, 'ondisk_log_size': 3786, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30987890, 'num_objects': 3783, 'num_object_clones': 0, 'num_object_copies': 15132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3784, 'num_write_kb': 30269, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:07:31.915 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3798", 'reported_seq': 3861, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:26.295530+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:07:26.295530+0000', 'last_peered': '2024-09-15T03:07:26.295530+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:07:26.295530+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3798, 'log_dups_size': 0, 'ondisk_log_size': 3798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31069810, 'num_objects': 3793, 'num_object_clones': 0, 'num_object_copies': 15172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3796, 'num_write_kb': 30356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1711}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:31.916 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:07:32.377 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 116 20 61980 61960 4.17183 2.22656 0.0274073 0.0374195 2024-09-15T03:07:33.372 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:33.372 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32009/246848 objects degraded (12.967%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":176,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61712,"data_bytes":505510460,"bytes_used":2119106560,"bytes_avail":384427950080,"bytes_total":386547056640,"degraded_objects":32009,"degraded_total":246848,"degraded_ratio":0.12967089058854031,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":83939,"recovering_keys_per_sec":0,"num_objects_recovered":123,"num_bytes_recovered":1007616,"num_keys_recovered":0,"write_bytes_sec":3297540,"read_op_per_sec":0,"write_op_per_sec":402},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:07:33.377 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 117 20 62286 62266 4.1566 2.39062 0.0799595 0.0375678 2024-09-15T03:07:33.430 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:07:33.430 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:34.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 118 20 62594 62574 4.14173 2.40625 0.0347126 0.0377109 2024-09-15T03:07:34.856 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:34.892 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-15T03:07:34.679631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505510460,"num_objects":61712,"num_object_clones":0,"num_object_copies":246848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61739,"num_write_kb":493794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":674,"num_bytes_recovered":5300224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61766,"ondisk_log_size":61766,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2069440,"kb_used_data":857760,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375417920,"statfs":{"total":386547056640,"available":384427950080,"internally_reserved":0,"allocated":878346240,"data_stored":877713474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,5,4,11,8,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26157056,"num_objects":3193,"num_object_clones":0,"num_object_copies":12772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":25544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003224"},"pg_stats":[{"pgid":"1.f","version":"31'3821","reported_seq":3863,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:30.860253+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:30.860253+0000","last_peered":"2024-09-15T03:07:30.860253+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:30.860253+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31301632,"num_objects":3821,"num_object_clones":0,"num_object_copies":15284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":30568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3703","reported_seq":3765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:31.288069+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:31.288069+0000","last_peered":"2024-09-15T03:07:31.288069+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:31.288069+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3703,"log_dups_size":0,"ondisk_log_size":3703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299762,"num_objects":3699,"num_object_clones":0,"num_object_copies":14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3701,"num_write_kb":29604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1701}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3949","reported_seq":4030,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:32.043802+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:32.043802+0000","last_peered":"2024-09-15T03:07:32.043802+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:32.043802+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32324409,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3948,"num_write_kb":31574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1844}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3702","reported_seq":3765,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:31.246473+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:31.246473+0000","last_peered":"2024-09-15T03:07:31.246473+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:31.246473+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291570,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3700,"num_write_kb":29589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1755}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3897","reported_seq":3952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:30.885864+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:30.885864+0000","last_peered":"2024-09-15T03:07:30.885864+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:30.885864+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31897202,"num_objects":3894,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":31157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1787}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3941","reported_seq":3984,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:30.945700+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:30.945700+0000","last_peered":"2024-09-15T03:07:30.945700+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:30.945700+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32284672,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3941,"num_write_kb":31528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3887","reported_seq":3952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:31.215319+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:31.215319+0000","last_peered":"2024-09-15T03:07:31.215319+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:31.215319+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3887,"log_dups_size":0,"ondisk_log_size":3887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31790706,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":31060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1829}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3925","reported_seq":3988,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:31.299662+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:31.299662+0000","last_peered":"2024-09-15T03:07:31.299662+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:31.299662+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32110194,"num_objects":3920,"num_object_clones":0,"num_object_copies":15680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":31372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1839}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3854","reported_seq":3911,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:30.908514+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:30.908514+0000","last_peered":"2024-09-15T03:07:30.908514+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:30.908514+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3854,"log_dups_size":0,"ondisk_log_size":3854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31501540,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3851,"num_write_kb":30785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3879","reported_seq":3962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:31.933345+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:31.933345+0000","last_peered":"2024-09-15T03:07:31.933345+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:31.933345+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3879,"log_dups_size":0,"ondisk_log_size":3879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757938,"num_objects":3877,"num_object_clones":0,"num_object_copies":15508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3877,"num_write_kb":31014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1819}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3914","reported_seq":3972,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:30.887207+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:30.887207+0000","last_peered":"2024-09-15T03:07:30.887207+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:30.887207+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32018859,"num_objects":3909,"num_object_clones":0,"num_object_copies":15636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3824","reported_seq":5156,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:30.978692+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:30.978692+0000","last_peered":"2024-09-15T03:07:30.978692+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:30.978692+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3824,"log_dups_size":0,"ondisk_log_size":3824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31316793,"num_objects":3823,"num_object_clones":0,"num_object_copies":15292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3823,"num_write_kb":30583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":648,"num_bytes_recovered":5300224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2478}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3874","reported_seq":3951,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:31.973167+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:31.973167+0000","last_peered":"2024-09-15T03:07:31.973167+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:31.973167+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3874,"log_dups_size":0,"ondisk_log_size":3874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31726393,"num_objects":3873,"num_object_clones":0,"num_object_copies":15492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3873,"num_write_kb":30983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3813","reported_seq":3861,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:30.972292+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:30.972292+0000","last_peered":"2024-09-15T03:07:30.972292+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:30.972292+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3813,"log_dups_size":0,"ondisk_log_size":3813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31200882,"num_objects":3809,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":30484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3889","reported_seq":3970,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:32.026407+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:32.026407+0000","last_peered":"2024-09-15T03:07:32.026407+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:32.026407+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3889,"log_dups_size":0,"ondisk_log_size":3889,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31831666,"num_objects":3886,"num_object_clones":0,"num_object_copies":15544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3887,"num_write_kb":31093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1898}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3894","reported_seq":3957,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:31.329860+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:31.329860+0000","last_peered":"2024-09-15T03:07:31.329860+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:31.329860+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31856242,"num_objects":3889,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":31124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1807}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":505510460,"num_objects":61712,"num_object_clones":0,"num_object_copies":246848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32009,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61739,"num_write_kb":493794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":674,"num_bytes_recovered":5300224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":880414720,"data_stored":880414720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61766,"ondisk_log_size":61766,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548232,"kb_used_data":246208,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93823608,"statfs":{"total":96636764160,"available":96075374592,"internally_reserved":0,"allocated":252116992,"data_stored":251958014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.694,"15min":1.694},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":16.027999999999999},{"interface":"front","average":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":8.2919999999999998,"5min":8.2919999999999998,"15min":8.2919999999999998},"last":0.55800000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":5.7409999999999997},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":5.7699999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:06:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.8600000000000001,"5min":1.8600000000000001,"15min":1.8600000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":3.3919999999999999,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.51900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548996,"kb_used_data":247484,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93822844,"statfs":{"total":96636764160,"available":96074592256,"internally_reserved":0,"allocated":253423616,"data_stored":253267784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,5,4,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.19700000000000001},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.223}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.25800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428632,"kb_used_data":118288,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93943208,"statfs":{"total":96636764160,"available":96197844992,"internally_reserved":0,"allocated":121126912,"data_stored":120967934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":1.425},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":5.1980000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":1.3340000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.503}]}]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543580,"kb_used_data":245780,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93828260,"statfs":{"total":96636764160,"available":96080138240,"internally_reserved":0,"allocated":251678720,"data_stored":251519742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.51700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":4.5190000000000001},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252653568,"data_stored":252653568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122064896,"data_stored":122064896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":253517824,"data_stored":253517824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252178432,"data_stored":252178432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:35.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 119 20 62907 62887 4.12747 2.44531 0.14294 0.0378376 2024-09-15T03:07:36.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:07:36.378909+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0379645 2024-09-15T03:07:36.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:07:36.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 120 20 63221 63201 4.11351 2.45312 0.0229942 0.0379645 2024-09-15T03:07:37.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 121 20 63534 63514 4.09973 2.44531 0.0338029 0.038091 2024-09-15T03:07:37.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:38.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 122 20 63849 63829 4.0863 2.46094 0.12296 0.0382194 2024-09-15T03:07:39.378 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:39.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 123 20 64148 64128 4.07207 2.33594 0.0839046 0.038353 2024-09-15T03:07:39.399 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:39.420 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-15T03:07:38.680779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":518183484,"num_objects":63259,"num_object_clones":0,"num_object_copies":253036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63286,"num_write_kb":506170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":717,"num_bytes_recovered":5652480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63313,"ondisk_log_size":63313,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2109292,"kb_used_data":885132,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375378068,"statfs":{"total":386547056640,"available":384387141632,"internally_reserved":0,"allocated":906375168,"data_stored":905742402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,10,2,9,14,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28991488,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":28312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"1.f","version":"31'3918","reported_seq":3958,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.919466+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:35.919466+0000","last_peered":"2024-09-15T03:07:35.919466+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:35.919466+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32096256,"num_objects":3918,"num_object_clones":0,"num_object_copies":15672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":31344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1942}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3793","reported_seq":3855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.352475+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:36.352475+0000","last_peered":"2024-09-15T03:07:36.352475+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:36.352475+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3793,"log_dups_size":0,"ondisk_log_size":3793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31037042,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3791,"num_write_kb":30324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4045","reported_seq":4126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:37.049596+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:37.049596+0000","last_peered":"2024-09-15T03:07:37.049596+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:37.049596+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33110841,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":32342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3795","reported_seq":3858,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.380578+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:36.380578+0000","last_peered":"2024-09-15T03:07:36.380578+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:36.380578+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31053426,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":30333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3994","reported_seq":4049,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.954670+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:35.954670+0000","last_peered":"2024-09-15T03:07:35.954670+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:35.954670+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3994,"log_dups_size":0,"ondisk_log_size":3994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691826,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3992,"num_write_kb":31933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4025","reported_seq":4069,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.916556+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:35.916556+0000","last_peered":"2024-09-15T03:07:35.916556+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:35.916556+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4025,"log_dups_size":0,"ondisk_log_size":4025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32972800,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":32200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4000","reported_seq":4065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.267966+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:36.267966+0000","last_peered":"2024-09-15T03:07:36.267966+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:36.267966+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4000,"log_dups_size":0,"ondisk_log_size":4000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32716402,"num_objects":3994,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":31964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1942}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4024","reported_seq":4087,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.208577+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:36.208577+0000","last_peered":"2024-09-15T03:07:36.208577+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:36.208577+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32921202,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":32164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3941","reported_seq":3998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.884921+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:35.884921+0000","last_peered":"2024-09-15T03:07:35.884921+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:35.884921+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32214244,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3938,"num_write_kb":31481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3982","reported_seq":4065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:37.079105+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:37.079105+0000","last_peered":"2024-09-15T03:07:37.079105+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:37.079105+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601714,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":31838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1922}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4006","reported_seq":4064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.872913+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:35.872913+0000","last_peered":"2024-09-15T03:07:35.872913+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:35.872913+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32772523,"num_objects":4001,"num_object_clones":0,"num_object_copies":16004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4003,"num_write_kb":32012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1923}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3912","reported_seq":5330,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.019643+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:36.019643+0000","last_peered":"2024-09-15T03:07:36.019643+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:36.019643+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32037689,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":5652480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2609}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3984","reported_seq":4061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:37.062711+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:37.062711+0000","last_peered":"2024-09-15T03:07:37.062711+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:37.062711+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32627513,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3983,"num_write_kb":31863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1944}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3911","reported_seq":3961,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.881164+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:35.881164+0000","last_peered":"2024-09-15T03:07:35.881164+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:35.881164+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32003698,"num_objects":3907,"num_object_clones":0,"num_object_copies":15628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3909,"num_write_kb":31268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1871}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3986","reported_seq":4067,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:37.044930+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:37.044930+0000","last_peered":"2024-09-15T03:07:37.044930+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:37.044930+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32626290,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3984,"num_write_kb":31869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1995}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3997","reported_seq":4060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.323866+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:36.323866+0000","last_peered":"2024-09-15T03:07:36.323866+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:36.323866+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32700018,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3995,"num_write_kb":31948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":518183484,"num_objects":63259,"num_object_clones":0,"num_object_copies":253036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63286,"num_write_kb":506170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":717,"num_bytes_recovered":5652480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":905854976,"data_stored":905854976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63313,"ondisk_log_size":63313,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562756,"kb_used_data":252412,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93809084,"statfs":{"total":96636764160,"available":96060502016,"internally_reserved":0,"allocated":258469888,"data_stored":258310910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,4,8,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":4.3520000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.437},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.371}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559572,"kb_used_data":253900,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93812268,"statfs":{"total":96636764160,"available":96063762432,"internally_reserved":0,"allocated":259993600,"data_stored":259837768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":8.3759999999999994},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.433}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.56299999999999994}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.372},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.39600000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427768,"kb_used_data":125744,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93944072,"statfs":{"total":96636764160,"available":96198729728,"internally_reserved":0,"allocated":128761856,"data_stored":128602878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.219},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":6.1150000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.251}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.36699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559196,"kb_used_data":253076,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93812644,"statfs":{"total":96636764160,"available":96064147456,"internally_reserved":0,"allocated":259149824,"data_stored":258990846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.192},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":1.764},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.46800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":1.728},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258961408,"data_stored":258961408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128532480,"data_stored":128532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259870720,"data_stored":259870720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258490368,"data_stored":258490368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:39.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:40.379 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 124 20 64448 64428 4.05813 2.34375 0.0417925 0.0384841 2024-09-15T03:07:40.416 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:40.424 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:40.434 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-15T03:07:38.680779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":518183484,"num_objects":63259,"num_object_clones":0,"num_object_copies":253036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63286,"num_write_kb":506170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":717,"num_bytes_recovered":5652480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63313,"ondisk_log_size":63313,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2109292,"kb_used_data":885132,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375378068,"statfs":{"total":386547056640,"available":384387141632,"internally_reserved":0,"allocated":906375168,"data_stored":905742402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,10,2,9,14,11,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28991488,"num_objects":3539,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-82,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":28312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"1.f","version":"31'3918","reported_seq":3958,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.919466+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:35.919466+0000","last_peered":"2024-09-15T03:07:35.919466+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:35.919466+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3918,"log_dups_size":0,"ondisk_log_size":3918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32096256,"num_objects":3918,"num_object_clones":0,"num_object_copies":15672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":31344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1942}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3793","reported_seq":3855,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.352475+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:36.352475+0000","last_peered":"2024-09-15T03:07:36.352475+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:36.352475+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3793,"log_dups_size":0,"ondisk_log_size":3793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31037042,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3791,"num_write_kb":30324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4045","reported_seq":4126,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:37.049596+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:37.049596+0000","last_peered":"2024-09-15T03:07:37.049596+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:37.049596+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33110841,"num_objects":4042,"num_object_clones":0,"num_object_copies":16168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":32342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3795","reported_seq":3858,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.380578+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:36.380578+0000","last_peered":"2024-09-15T03:07:36.380578+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:36.380578+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31053426,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":30333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3994","reported_seq":4049,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.954670+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:35.954670+0000","last_peered":"2024-09-15T03:07:35.954670+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:35.954670+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3994,"log_dups_size":0,"ondisk_log_size":3994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32691826,"num_objects":3991,"num_object_clones":0,"num_object_copies":15964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3992,"num_write_kb":31933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4025","reported_seq":4069,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.916556+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:35.916556+0000","last_peered":"2024-09-15T03:07:35.916556+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:35.916556+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4025,"log_dups_size":0,"ondisk_log_size":4025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32972800,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4025,"num_write_kb":32200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4000","reported_seq":4065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.267966+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:36.267966+0000","last_peered":"2024-09-15T03:07:36.267966+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:36.267966+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4000,"log_dups_size":0,"ondisk_log_size":4000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32716402,"num_objects":3994,"num_object_clones":0,"num_object_copies":15976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3998,"num_write_kb":31964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1942}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4024","reported_seq":4087,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.208577+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:36.208577+0000","last_peered":"2024-09-15T03:07:36.208577+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:36.208577+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32921202,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4022,"num_write_kb":32164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1938}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3941","reported_seq":3998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.884921+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:35.884921+0000","last_peered":"2024-09-15T03:07:35.884921+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:35.884921+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32214244,"num_objects":3933,"num_object_clones":0,"num_object_copies":15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3938,"num_write_kb":31481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3982","reported_seq":4065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:37.079105+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:37.079105+0000","last_peered":"2024-09-15T03:07:37.079105+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:37.079105+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601714,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":31838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1922}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4006","reported_seq":4064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.872913+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:35.872913+0000","last_peered":"2024-09-15T03:07:35.872913+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:35.872913+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32772523,"num_objects":4001,"num_object_clones":0,"num_object_copies":16004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4003,"num_write_kb":32012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1923}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3912","reported_seq":5330,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.019643+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:36.019643+0000","last_peered":"2024-09-15T03:07:36.019643+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:36.019643+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3912,"log_dups_size":0,"ondisk_log_size":3912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32037689,"num_objects":3911,"num_object_clones":0,"num_object_copies":15644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3911,"num_write_kb":31287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":5652480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2609}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3984","reported_seq":4061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:37.062711+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:37.062711+0000","last_peered":"2024-09-15T03:07:37.062711+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:37.062711+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32627513,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3983,"num_write_kb":31863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1944}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3911","reported_seq":3961,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:35.881164+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:35.881164+0000","last_peered":"2024-09-15T03:07:35.881164+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:35.881164+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3911,"log_dups_size":0,"ondisk_log_size":3911,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32003698,"num_objects":3907,"num_object_clones":0,"num_object_copies":15628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3909,"num_write_kb":31268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1871}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3986","reported_seq":4067,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:37.044930+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:37.044930+0000","last_peered":"2024-09-15T03:07:37.044930+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:37.044930+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32626290,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3984,"num_write_kb":31869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1995}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3997","reported_seq":4060,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:36.323866+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:36.323866+0000","last_peered":"2024-09-15T03:07:36.323866+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:36.323866+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32700018,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3995,"num_write_kb":31948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":518183484,"num_objects":63259,"num_object_clones":0,"num_object_copies":253036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63286,"num_write_kb":506170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":717,"num_bytes_recovered":5652480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":905854976,"data_stored":905854976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63313,"ondisk_log_size":63313,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562756,"kb_used_data":252412,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93809084,"statfs":{"total":96636764160,"available":96060502016,"internally_reserved":0,"allocated":258469888,"data_stored":258310910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,4,8,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":4.3520000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.437},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.41699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.371}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559572,"kb_used_data":253900,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93812268,"statfs":{"total":96636764160,"available":96063762432,"internally_reserved":0,"allocated":259993600,"data_stored":259837768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":8.3759999999999994},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.433}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.56299999999999994}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.372},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.39600000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":427768,"kb_used_data":125744,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93944072,"statfs":{"total":96636764160,"available":96198729728,"internally_reserved":0,"allocated":128761856,"data_stored":128602878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.219},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":6.1150000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.251}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.36699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559196,"kb_used_data":253076,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93812644,"statfs":{"total":96636764160,"available":96064147456,"internally_reserved":0,"allocated":259149824,"data_stored":258990846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.192},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":1.764},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.46800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":1.728},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258961408,"data_stored":258961408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128532480,"data_stored":128532480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259870720,"data_stored":259870720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258490368,"data_stored":258490368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:40.435 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:07:40.435 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3918", 'reported_seq': 3958, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:35.919466+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:07:35.919466+0000', 'last_peered': '2024-09-15T03:07:35.919466+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:07:35.919466+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3918, 'log_dups_size': 0, 'ondisk_log_size': 3918, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32096256, 'num_objects': 3918, 'num_object_clones': 0, 'num_object_copies': 15672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3918, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3918, 'num_write_kb': 31344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1942}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:40.435 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3793", 'reported_seq': 3855, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:36.352475+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:07:36.352475+0000', 'last_peered': '2024-09-15T03:07:36.352475+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:07:36.352475+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3793, 'log_dups_size': 0, 'ondisk_log_size': 3793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31037042, 'num_objects': 3789, 'num_object_clones': 0, 'num_object_copies': 15156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3791, 'num_write_kb': 30324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4045", 'reported_seq': 4126, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:37.049596+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:07:37.049596+0000', 'last_peered': '2024-09-15T03:07:37.049596+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:07:37.049596+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4045, 'log_dups_size': 0, 'ondisk_log_size': 4045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33110841, 'num_objects': 4042, 'num_object_clones': 0, 'num_object_copies': 16168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4044, 'num_write_kb': 32342, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1940}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3795", 'reported_seq': 3858, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:36.380578+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:07:36.380578+0000', 'last_peered': '2024-09-15T03:07:36.380578+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:07:36.380578+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3795, 'log_dups_size': 0, 'ondisk_log_size': 3795, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31053426, 'num_objects': 3791, 'num_object_clones': 0, 'num_object_copies': 15164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3793, 'num_write_kb': 30333, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1848}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3994", 'reported_seq': 4049, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:35.954670+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:07:35.954670+0000', 'last_peered': '2024-09-15T03:07:35.954670+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:07:35.954670+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3994, 'log_dups_size': 0, 'ondisk_log_size': 3994, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32691826, 'num_objects': 3991, 'num_object_clones': 0, 'num_object_copies': 15964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3992, 'num_write_kb': 31933, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1884}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4025", 'reported_seq': 4069, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:35.916556+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:07:35.916556+0000', 'last_peered': '2024-09-15T03:07:35.916556+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:07:35.916556+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4025, 'log_dups_size': 0, 'ondisk_log_size': 4025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32972800, 'num_objects': 4025, 'num_object_clones': 0, 'num_object_copies': 16100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4025, 'num_write_kb': 32200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1969}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:40.436 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4000", 'reported_seq': 4065, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:36.267966+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:07:36.267966+0000', 'last_peered': '2024-09-15T03:07:36.267966+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:07:36.267966+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4000, 'log_dups_size': 0, 'ondisk_log_size': 4000, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32716402, 'num_objects': 3994, 'num_object_clones': 0, 'num_object_copies': 15976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3998, 'num_write_kb': 31964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1942}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4024", 'reported_seq': 4087, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:36.208577+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:07:36.208577+0000', 'last_peered': '2024-09-15T03:07:36.208577+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:07:36.208577+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4024, 'log_dups_size': 0, 'ondisk_log_size': 4024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32921202, 'num_objects': 4019, 'num_object_clones': 0, 'num_object_copies': 16076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4022, 'num_write_kb': 32164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1938}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3941", 'reported_seq': 3998, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:35.884921+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:07:35.884921+0000', 'last_peered': '2024-09-15T03:07:35.884921+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:07:35.884921+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3941, 'log_dups_size': 0, 'ondisk_log_size': 3941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32214244, 'num_objects': 3933, 'num_object_clones': 0, 'num_object_copies': 15732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3933, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3938, 'num_write_kb': 31481, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1891}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3982", 'reported_seq': 4065, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:37.079105+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:07:37.079105+0000', 'last_peered': '2024-09-15T03:07:37.079105+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:07:37.079105+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3982, 'log_dups_size': 0, 'ondisk_log_size': 3982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32601714, 'num_objects': 3980, 'num_object_clones': 0, 'num_object_copies': 15920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3980, 'num_write_kb': 31838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1922}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4006", 'reported_seq': 4064, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:35.872913+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:07:35.872913+0000', 'last_peered': '2024-09-15T03:07:35.872913+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:07:35.872913+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4006, 'log_dups_size': 0, 'ondisk_log_size': 4006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32772523, 'num_objects': 4001, 'num_object_clones': 0, 'num_object_copies': 16004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4003, 'num_write_kb': 32012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1923}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:40.437 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3912", 'reported_seq': 5330, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:36.019643+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:07:36.019643+0000', 'last_peered': '2024-09-15T03:07:36.019643+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:07:36.019643+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3912, 'log_dups_size': 0, 'ondisk_log_size': 3912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32037689, 'num_objects': 3911, 'num_object_clones': 0, 'num_object_copies': 15644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1302, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3911, 'num_write_kb': 31287, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 691, 'num_bytes_recovered': 5652480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2609}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3984", 'reported_seq': 4061, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:37.062711+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:07:37.062711+0000', 'last_peered': '2024-09-15T03:07:37.062711+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:07:37.062711+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3984, 'log_dups_size': 0, 'ondisk_log_size': 3984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32627513, 'num_objects': 3983, 'num_object_clones': 0, 'num_object_copies': 15932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3983, 'num_write_kb': 31863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1944}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3911", 'reported_seq': 3961, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:35.881164+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:07:35.881164+0000', 'last_peered': '2024-09-15T03:07:35.881164+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:07:35.881164+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3911, 'log_dups_size': 0, 'ondisk_log_size': 3911, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32003698, 'num_objects': 3907, 'num_object_clones': 0, 'num_object_copies': 15628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3909, 'num_write_kb': 31268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3986", 'reported_seq': 4067, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:37.044930+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:07:37.044930+0000', 'last_peered': '2024-09-15T03:07:37.044930+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:07:37.044930+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3986, 'log_dups_size': 0, 'ondisk_log_size': 3986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32626290, 'num_objects': 3983, 'num_object_clones': 0, 'num_object_copies': 15932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3984, 'num_write_kb': 31869, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1995}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:07:40.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3997", 'reported_seq': 4060, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:36.323866+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:07:36.323866+0000', 'last_peered': '2024-09-15T03:07:36.323866+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:07:36.323866+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3997, 'log_dups_size': 0, 'ondisk_log_size': 3997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32700018, 'num_objects': 3992, 'num_object_clones': 0, 'num_object_copies': 15968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3995, 'num_write_kb': 31948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:40.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:07:41.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 125 20 64739 64719 4.04385 2.27344 0.14164 0.0386129 2024-09-15T03:07:41.823 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:41.824 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31966/253036 objects degraded (12.633%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":184,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63259,"data_bytes":518183484,"bytes_used":2159915008,"bytes_avail":384387141632,"bytes_total":386547056640,"degraded_objects":31966,"degraded_total":253036,"degraded_ratio":0.12632985029798133,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":55957,"recovering_keys_per_sec":0,"num_objects_recovered":82,"num_bytes_recovered":671744,"num_keys_recovered":0,"write_bytes_sec":2146167,"read_op_per_sec":0,"write_op_per_sec":261},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:07:41.937 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:07:41.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:42.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 126 20 65025 65005 4.02949 2.23438 0.0219773 0.0387614 2024-09-15T03:07:43.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 127 20 65315 65295 4.01561 2.26562 0.0226975 0.0388957 2024-09-15T03:07:43.437 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:43.481 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:43.487 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-15T03:07:42.683074+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530561596,"num_objects":64770,"num_object_clones":0,"num_object_copies":259080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64797,"num_write_kb":518258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":753,"num_bytes_recovered":5947392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64824,"ondisk_log_size":64824,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2137716,"kb_used_data":909396,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375349644,"statfs":{"total":386547056640,"available":384358035456,"internally_reserved":0,"allocated":931221504,"data_stored":930588738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,12,4,6,8,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38141952,"num_objects":4656,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":37248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004523"},"pg_stats":[{"pgid":"1.f","version":"31'4017","reported_seq":4058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.130035+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:41.130035+0000","last_peered":"2024-09-15T03:07:41.130035+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:41.130035+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32907264,"num_objects":4017,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4017,"num_write_kb":32136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3902","reported_seq":3964,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.369082+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:41.369082+0000","last_peered":"2024-09-15T03:07:41.369082+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:41.369082+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3902,"log_dups_size":0,"ondisk_log_size":3902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31929970,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3900,"num_write_kb":31196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4140","reported_seq":4221,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:42.097822+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:42.097822+0000","last_peered":"2024-09-15T03:07:42.097822+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:42.097822+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33889081,"num_objects":4137,"num_object_clones":0,"num_object_copies":16548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4139,"num_write_kb":33102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2035}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3895","reported_seq":3958,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.231958+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:41.231958+0000","last_peered":"2024-09-15T03:07:41.231958+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:41.231958+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3895,"log_dups_size":0,"ondisk_log_size":3895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872626,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3893,"num_write_kb":31133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1948}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4089","reported_seq":4144,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.103591+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:41.103591+0000","last_peered":"2024-09-15T03:07:41.103591+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:41.103591+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4089,"log_dups_size":0,"ondisk_log_size":4089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33470066,"num_objects":4086,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1979}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4106","reported_seq":4150,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.113197+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:41.113197+0000","last_peered":"2024-09-15T03:07:41.113197+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:41.113197+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33636352,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4106,"num_write_kb":32848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2050}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4095","reported_seq":4160,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.348124+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:41.348124+0000","last_peered":"2024-09-15T03:07:41.348124+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:41.348124+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4095,"log_dups_size":0,"ondisk_log_size":4095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33494642,"num_objects":4089,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":32724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2037}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4128","reported_seq":4191,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.282651+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:41.282651+0000","last_peered":"2024-09-15T03:07:41.282651+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:41.282651+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33773170,"num_objects":4123,"num_object_clones":0,"num_object_copies":16492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4126,"num_write_kb":32996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4038","reported_seq":4095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:40.981900+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:40.981900+0000","last_peered":"2024-09-15T03:07:40.981900+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:40.981900+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33008868,"num_objects":4030,"num_object_clones":0,"num_object_copies":16120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4035,"num_write_kb":32257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4069","reported_seq":4152,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.967146+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:41.967146+0000","last_peered":"2024-09-15T03:07:41.967146+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:41.967146+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33314418,"num_objects":4067,"num_object_clones":0,"num_object_copies":16268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4067,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2009}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4090","reported_seq":4148,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:40.773333+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:40.773333+0000","last_peered":"2024-09-15T03:07:40.773333+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:40.773333+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33460651,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2007}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4014","reported_seq":5504,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.071983+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:41.071983+0000","last_peered":"2024-09-15T03:07:41.071983+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:41.071983+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32873273,"num_objects":4013,"num_object_clones":0,"num_object_copies":16052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4013,"num_write_kb":32103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":727,"num_bytes_recovered":5947392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2747}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4081","reported_seq":4158,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:42.003552+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:42.003552+0000","last_peered":"2024-09-15T03:07:42.003552+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:42.003552+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4081,"log_dups_size":0,"ondisk_log_size":4081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33422137,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4080,"num_write_kb":32639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2041}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4003","reported_seq":4053,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:40.985861+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:40.985861+0000","last_peered":"2024-09-15T03:07:40.985861+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:40.985861+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32757362,"num_objects":3999,"num_object_clones":0,"num_object_copies":15996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4001,"num_write_kb":32004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4076","reported_seq":4156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:42.017178+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:42.017178+0000","last_peered":"2024-09-15T03:07:42.017178+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:42.017178+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33363570,"num_objects":4073,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":32589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2085}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4081","reported_seq":4144,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.246912+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:41.246912+0000","last_peered":"2024-09-15T03:07:41.246912+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:41.246912+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4081,"log_dups_size":0,"ondisk_log_size":4081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33388146,"num_objects":4076,"num_object_clones":0,"num_object_copies":16304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4079,"num_write_kb":32620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530561596,"num_objects":64770,"num_object_clones":0,"num_object_copies":259080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64797,"num_write_kb":518258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":753,"num_bytes_recovered":5947392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":930988032,"data_stored":930988032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64824,"ondisk_log_size":64824,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556400,"kb_used_data":258536,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93815440,"statfs":{"total":96636764160,"available":96067010560,"internally_reserved":0,"allocated":264740864,"data_stored":264581886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,2,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":2.5750000000000002},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.193}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":1.619},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":2.528}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":2.8519999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573840,"kb_used_data":259848,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93798000,"statfs":{"total":96636764160,"available":96049152000,"internally_reserved":0,"allocated":266084352,"data_stored":265928520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.43099999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.39300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":3.2690000000000001},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.45900000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":438064,"kb_used_data":131880,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93933776,"statfs":{"total":96636764160,"available":96188186624,"internally_reserved":0,"allocated":135045120,"data_stored":134886142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":3.6640000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":3.5910000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":2.6829999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.6389999999999998},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.31900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569412,"kb_used_data":259132,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93802428,"statfs":{"total":96636764160,"available":96053686272,"internally_reserved":0,"allocated":265351168,"data_stored":265192190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":3.036}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":3.0920000000000001},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.45600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":2.7799999999999998},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":5.1970000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265203712,"data_stored":265203712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134889472,"data_stored":134889472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266014720,"data_stored":266014720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":264880128,"data_stored":264880128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:44.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 128 20 65616 65596 4.00261 2.35156 0.0282888 0.0390187 2024-09-15T03:07:45.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 129 20 65928 65908 3.99047 2.4375 0.0311254 0.0391432 2024-09-15T03:07:46.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 130 20 66232 66212 3.97805 2.375 0.0172771 0.0392543 2024-09-15T03:07:46.489 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:47.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 131 20 66524 66504 3.9651 2.28125 0.0160116 0.0393866 2024-09-15T03:07:47.968 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:47.981 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:48.009 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-15T03:07:46.684128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539949628,"num_objects":65916,"num_object_clones":0,"num_object_copies":263664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65943,"num_write_kb":527426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65970,"ondisk_log_size":65970,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2149752,"kb_used_data":921432,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375337608,"statfs":{"total":386547056640,"available":384345710592,"internally_reserved":0,"allocated":943546368,"data_stored":942913602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,4,12,7,8,6,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34439168,"num_objects":4204,"num_object_clones":0,"num_object_copies":16816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":33632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004496"},"pg_stats":[{"pgid":"1.f","version":"31'4127","reported_seq":4169,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:45.974657+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:45.974657+0000","last_peered":"2024-09-15T03:07:45.974657+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:45.974657+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2151}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3997","reported_seq":4059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.330542+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:46.330542+0000","last_peered":"2024-09-15T03:07:46.330542+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:46.330542+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32708210,"num_objects":3993,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3995,"num_write_kb":31956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1995}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4140","reported_seq":4221,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:42.097822+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:42.097822+0000","last_peered":"2024-09-15T03:07:42.097822+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:42.097822+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33889081,"num_objects":4137,"num_object_clones":0,"num_object_copies":16548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4139,"num_write_kb":33102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2035}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3984","reported_seq":4047,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.397457+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:46.397457+0000","last_peered":"2024-09-15T03:07:46.397457+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:46.397457+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601714,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":31845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4201","reported_seq":4257,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:45.999952+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:45.999952+0000","last_peered":"2024-09-15T03:07:45.999952+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:45.999952+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387570,"num_objects":4198,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":33589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4199","reported_seq":4242,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.167915+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:46.167915+0000","last_peered":"2024-09-15T03:07:46.167915+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:46.167915+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34398208,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":33592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4186","reported_seq":4251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.373357+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:46.373357+0000","last_peered":"2024-09-15T03:07:46.373357+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:46.373357+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34240114,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":33452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2128}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4212","reported_seq":4275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.331816+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:46.331816+0000","last_peered":"2024-09-15T03:07:46.331816+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:46.331816+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34461298,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4135","reported_seq":4190,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.118426+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:46.118426+0000","last_peered":"2024-09-15T03:07:46.118426+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:46.118426+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33803492,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":33033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2085}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4069","reported_seq":4152,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:41.967146+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:41.967146+0000","last_peered":"2024-09-15T03:07:41.967146+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:41.967146+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33314418,"num_objects":4067,"num_object_clones":0,"num_object_copies":16268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4067,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2009}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4182","reported_seq":4240,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.053142+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:46.053142+0000","last_peered":"2024-09-15T03:07:46.053142+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:46.053142+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34214315,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":33420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4105","reported_seq":5662,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.103102+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:46.103102+0000","last_peered":"2024-09-15T03:07:46.103102+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:46.103102+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33618745,"num_objects":4104,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4104,"num_write_kb":32831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4081","reported_seq":4158,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:42.003552+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:42.003552+0000","last_peered":"2024-09-15T03:07:42.003552+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:42.003552+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4081,"log_dups_size":0,"ondisk_log_size":4081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33422137,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4080,"num_write_kb":32639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2041}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4085","reported_seq":4135,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.021596+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:46.021596+0000","last_peered":"2024-09-15T03:07:46.021596+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:46.021596+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33429106,"num_objects":4081,"num_object_clones":0,"num_object_copies":16324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":32660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2045}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4076","reported_seq":4156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:42.017178+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:42.017178+0000","last_peered":"2024-09-15T03:07:42.017178+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:42.017178+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33363570,"num_objects":4073,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":32589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2085}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4191","reported_seq":4252,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.434178+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:46.434178+0000","last_peered":"2024-09-15T03:07:46.434178+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:46.434178+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34289266,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":33500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":539949628,"num_objects":65916,"num_object_clones":0,"num_object_copies":263664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65943,"num_write_kb":527426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":943308800,"data_stored":943308800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65970,"ondisk_log_size":65970,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570768,"kb_used_data":264584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93801072,"statfs":{"total":96636764160,"available":96052297728,"internally_reserved":0,"allocated":270934016,"data_stored":270775038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,0,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.355},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.55900000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.44900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.48199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573840,"kb_used_data":259848,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93798000,"statfs":{"total":96636764160,"available":96049152000,"internally_reserved":0,"allocated":266084352,"data_stored":265928520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.43099999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.39300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":3.2690000000000001},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.45900000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":438064,"kb_used_data":131880,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93933776,"statfs":{"total":96636764160,"available":96188186624,"internally_reserved":0,"allocated":135045120,"data_stored":134886142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":3.6640000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":3.5910000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":2.6829999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.6389999999999998},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.31900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567080,"kb_used_data":265120,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93804760,"statfs":{"total":96636764160,"available":96056074240,"internally_reserved":0,"allocated":271482880,"data_stored":271323902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,3,5,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271392768,"data_stored":271392768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":134889472,"data_stored":134889472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266014720,"data_stored":266014720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271011840,"data_stored":271011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:48.010 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:48.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 132 20 66818 66798 3.95246 2.29688 0.0113796 0.0395105 2024-09-15T03:07:49.312 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:49.325 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:49.344 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-15T03:07:48.684679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542931516,"num_objects":66280,"num_object_clones":0,"num_object_copies":265120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66307,"num_write_kb":530338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66334,"ondisk_log_size":66334,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2153392,"kb_used_data":933392,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375333968,"statfs":{"total":386547056640,"available":384341983232,"internally_reserved":0,"allocated":955793408,"data_stored":955160642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,14,6,15,7,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28073984,"num_objects":3427,"num_object_clones":0,"num_object_copies":13708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-70,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3427,"num_write_kb":27416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004445"},"pg_stats":[{"pgid":"1.f","version":"31'4127","reported_seq":4169,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:45.974657+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:45.974657+0000","last_peered":"2024-09-15T03:07:45.974657+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:45.974657+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2151}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3997","reported_seq":4059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.330542+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:46.330542+0000","last_peered":"2024-09-15T03:07:46.330542+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:46.330542+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32708210,"num_objects":3993,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3995,"num_write_kb":31956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1995}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4242","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:47.107276+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:47.107276+0000","last_peered":"2024-09-15T03:07:47.107276+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:47.107276+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34724665,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":33918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2137}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3984","reported_seq":4047,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.397457+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:46.397457+0000","last_peered":"2024-09-15T03:07:46.397457+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:46.397457+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601714,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":31845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4201","reported_seq":4257,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:45.999952+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:45.999952+0000","last_peered":"2024-09-15T03:07:45.999952+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:45.999952+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387570,"num_objects":4198,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":33589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4199","reported_seq":4242,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.167915+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:46.167915+0000","last_peered":"2024-09-15T03:07:46.167915+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:46.167915+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34398208,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":33592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4186","reported_seq":4251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.373357+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:46.373357+0000","last_peered":"2024-09-15T03:07:46.373357+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:46.373357+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34240114,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":33452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2128}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4212","reported_seq":4275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.331816+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:46.331816+0000","last_peered":"2024-09-15T03:07:46.331816+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:46.331816+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34461298,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4135","reported_seq":4190,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.118426+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:46.118426+0000","last_peered":"2024-09-15T03:07:46.118426+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:46.118426+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33803492,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":33033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2085}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4164","reported_seq":4247,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.892475+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:46.892475+0000","last_peered":"2024-09-15T03:07:46.892475+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:46.892475+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34092658,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4162,"num_write_kb":33294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4182","reported_seq":4240,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.053142+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:46.053142+0000","last_peered":"2024-09-15T03:07:46.053142+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:46.053142+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34214315,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":33420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4105","reported_seq":5662,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.103102+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:46.103102+0000","last_peered":"2024-09-15T03:07:46.103102+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:46.103102+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33618745,"num_objects":4104,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4104,"num_write_kb":32831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4175","reported_seq":4252,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.822315+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:46.822315+0000","last_peered":"2024-09-15T03:07:46.822315+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:46.822315+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34192185,"num_objects":4174,"num_object_clones":0,"num_object_copies":16696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4174,"num_write_kb":33391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2135}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4085","reported_seq":4135,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.021596+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:46.021596+0000","last_peered":"2024-09-15T03:07:46.021596+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:46.021596+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33429106,"num_objects":4081,"num_object_clones":0,"num_object_copies":16324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":32660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2045}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4149","reported_seq":4230,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.962329+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:46.962329+0000","last_peered":"2024-09-15T03:07:46.962329+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:46.962329+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33961586,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2158}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4191","reported_seq":4252,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.434178+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:46.434178+0000","last_peered":"2024-09-15T03:07:46.434178+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:46.434178+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34289266,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":33500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":542931516,"num_objects":66280,"num_object_clones":0,"num_object_copies":265120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66307,"num_write_kb":530338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":955846656,"data_stored":955846656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66334,"ondisk_log_size":66334,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570768,"kb_used_data":264584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93801072,"statfs":{"total":96636764160,"available":96052297728,"internally_reserved":0,"allocated":270934016,"data_stored":270775038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,0,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.355},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.55900000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.44900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.48199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567220,"kb_used_data":265708,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93804620,"statfs":{"total":96636764160,"available":96055930880,"internally_reserved":0,"allocated":272084992,"data_stored":271929160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,0,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.29899999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.502}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.33600000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448324,"kb_used_data":137980,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93923516,"statfs":{"total":96636764160,"available":96177680384,"internally_reserved":0,"allocated":141291520,"data_stored":141132542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.222},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.47299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.253}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567080,"kb_used_data":265120,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93804760,"statfs":{"total":96636764160,"available":96056074240,"internally_reserved":0,"allocated":271482880,"data_stored":271323902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,3,5,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271392768,"data_stored":271392768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141262848,"data_stored":141262848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272179200,"data_stored":272179200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271011840,"data_stored":271011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:49.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:07:49.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4127", 'reported_seq': 4169, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:45.974657+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:07:45.974657+0000', 'last_peered': '2024-09-15T03:07:45.974657+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:07:45.974657+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4127, 'log_dups_size': 0, 'ondisk_log_size': 4127, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33808384, 'num_objects': 4127, 'num_object_clones': 0, 'num_object_copies': 16508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4127, 'num_write_kb': 33016, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2151}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:49.345 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:07:49.345 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3997", 'reported_seq': 4059, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.330542+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:07:46.330542+0000', 'last_peered': '2024-09-15T03:07:46.330542+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:07:46.330542+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3997, 'log_dups_size': 0, 'ondisk_log_size': 3997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32708210, 'num_objects': 3993, 'num_object_clones': 0, 'num_object_copies': 15972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3995, 'num_write_kb': 31956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1995}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4242", 'reported_seq': 4322, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:47.107276+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:07:47.107276+0000', 'last_peered': '2024-09-15T03:07:47.107276+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:07:47.107276+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4242, 'log_dups_size': 0, 'ondisk_log_size': 4242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34724665, 'num_objects': 4239, 'num_object_clones': 0, 'num_object_copies': 16956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4241, 'num_write_kb': 33918, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2137}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3984", 'reported_seq': 4047, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.397457+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:07:46.397457+0000', 'last_peered': '2024-09-15T03:07:46.397457+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:07:46.397457+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 3984, 'log_dups_size': 0, 'ondisk_log_size': 3984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32601714, 'num_objects': 3980, 'num_object_clones': 0, 'num_object_copies': 15920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3982, 'num_write_kb': 31845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4201", 'reported_seq': 4257, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:45.999952+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:07:45.999952+0000', 'last_peered': '2024-09-15T03:07:45.999952+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:07:45.999952+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4201, 'log_dups_size': 0, 'ondisk_log_size': 4201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34387570, 'num_objects': 4198, 'num_object_clones': 0, 'num_object_copies': 16792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4198, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4199, 'num_write_kb': 33589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2091}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4199", 'reported_seq': 4242, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.167915+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:07:46.167915+0000', 'last_peered': '2024-09-15T03:07:46.167915+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:07:46.167915+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4199, 'log_dups_size': 0, 'ondisk_log_size': 4199, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34398208, 'num_objects': 4199, 'num_object_clones': 0, 'num_object_copies': 16796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4199, 'num_write_kb': 33592, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2143}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:07:49.346 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4186", 'reported_seq': 4251, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.373357+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:07:46.373357+0000', 'last_peered': '2024-09-15T03:07:46.373357+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:07:46.373357+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4186, 'log_dups_size': 0, 'ondisk_log_size': 4186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34240114, 'num_objects': 4180, 'num_object_clones': 0, 'num_object_copies': 16720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4184, 'num_write_kb': 33452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2128}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:49.347 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:07:49.347 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4212", 'reported_seq': 4275, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.331816+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:07:46.331816+0000', 'last_peered': '2024-09-15T03:07:46.331816+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:07:46.331816+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4212, 'log_dups_size': 0, 'ondisk_log_size': 4212, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34461298, 'num_objects': 4207, 'num_object_clones': 0, 'num_object_copies': 16828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4210, 'num_write_kb': 33668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:49.347 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:07:49.347 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4135", 'reported_seq': 4190, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.118426+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:07:46.118426+0000', 'last_peered': '2024-09-15T03:07:46.118426+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:07:46.118426+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4135, 'log_dups_size': 0, 'ondisk_log_size': 4135, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33803492, 'num_objects': 4127, 'num_object_clones': 0, 'num_object_copies': 16508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4132, 'num_write_kb': 33033, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2085}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:49.347 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:07:49.347 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4164", 'reported_seq': 4247, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.892475+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:07:46.892475+0000', 'last_peered': '2024-09-15T03:07:46.892475+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:07:46.892475+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4164, 'log_dups_size': 0, 'ondisk_log_size': 4164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34092658, 'num_objects': 4162, 'num_object_clones': 0, 'num_object_copies': 16648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4162, 'num_write_kb': 33294, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2104}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:49.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:07:49.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4182", 'reported_seq': 4240, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.053142+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:07:46.053142+0000', 'last_peered': '2024-09-15T03:07:46.053142+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:07:46.053142+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4182, 'log_dups_size': 0, 'ondisk_log_size': 4182, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34214315, 'num_objects': 4177, 'num_object_clones': 0, 'num_object_copies': 16708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4179, 'num_write_kb': 33420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2099}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:49.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:07:49.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4105", 'reported_seq': 5662, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.103102+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:07:46.103102+0000', 'last_peered': '2024-09-15T03:07:46.103102+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:07:46.103102+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4105, 'log_dups_size': 0, 'ondisk_log_size': 4105, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33618745, 'num_objects': 4104, 'num_object_clones': 0, 'num_object_copies': 16416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4104, 'num_write_kb': 32831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 761, 'num_bytes_recovered': 6225920, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:49.348 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:07:49.348 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4175", 'reported_seq': 4252, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.822315+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:07:46.822315+0000', 'last_peered': '2024-09-15T03:07:46.822315+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:07:46.822315+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4175, 'log_dups_size': 0, 'ondisk_log_size': 4175, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34192185, 'num_objects': 4174, 'num_object_clones': 0, 'num_object_copies': 16696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4174, 'num_write_kb': 33391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:49.349 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:07:49.349 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4085", 'reported_seq': 4135, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.021596+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:07:46.021596+0000', 'last_peered': '2024-09-15T03:07:46.021596+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:07:46.021596+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4085, 'log_dups_size': 0, 'ondisk_log_size': 4085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33429106, 'num_objects': 4081, 'num_object_clones': 0, 'num_object_copies': 16324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4083, 'num_write_kb': 32660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:49.349 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:07:49.349 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4149", 'reported_seq': 4230, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.962329+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:07:46.962329+0000', 'last_peered': '2024-09-15T03:07:46.962329+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:07:46.962329+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4149, 'log_dups_size': 0, 'ondisk_log_size': 4149, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33961586, 'num_objects': 4146, 'num_object_clones': 0, 'num_object_copies': 16584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4147, 'num_write_kb': 33173, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2158}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:49.350 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:07:49.350 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4191", 'reported_seq': 4252, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:46.434178+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:07:46.434178+0000', 'last_peered': '2024-09-15T03:07:46.434178+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:07:46.434178+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4191, 'log_dups_size': 0, 'ondisk_log_size': 4191, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34289266, 'num_objects': 4186, 'num_object_clones': 0, 'num_object_copies': 16744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4189, 'num_write_kb': 33500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:49.350 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:07:49.380 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 133 20 67117 67097 3.9403 2.33594 0.0395212 0.0396378 2024-09-15T03:07:50.381 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 134 20 67408 67388 3.92784 2.27344 0.0977011 0.0397648 2024-09-15T03:07:51.005 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:51.005 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31896/265120 objects degraded (12.031%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":194,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66280,"data_bytes":542931516,"bytes_used":2205073408,"bytes_avail":384341983232,"bytes_total":386547056640,"degraded_objects":31896,"degraded_total":265120,"degraded_ratio":0.12030778515389258,"recovering_objects_per_sec":5,"recovering_bytes_per_sec":47760,"recovering_keys_per_sec":0,"num_objects_recovered":70,"num_bytes_recovered":573440,"num_keys_recovered":0,"write_bytes_sec":2061200,"read_op_per_sec":0,"write_op_per_sec":251},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:07:51.073 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:07:51.073 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:51.381 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 135 20 67704 67684 3.91588 2.3125 0.0778741 0.0398846 2024-09-15T03:07:51.931 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:51.948 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:51.956 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-15T03:07:50.687392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542931516,"num_objects":66280,"num_object_clones":0,"num_object_copies":265120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66307,"num_write_kb":530338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66334,"ondisk_log_size":66334,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2153392,"kb_used_data":933392,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375333968,"statfs":{"total":386547056640,"available":384341983232,"internally_reserved":0,"allocated":955793408,"data_stored":955160642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,14,6,15,7,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24748032,"num_objects":3021,"num_object_clones":0,"num_object_copies":12084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-70,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3021,"num_write_kb":24168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":573440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006613"},"pg_stats":[{"pgid":"1.f","version":"31'4127","reported_seq":4169,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:45.974657+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:45.974657+0000","last_peered":"2024-09-15T03:07:45.974657+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:45.974657+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33808384,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":33016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2151}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3997","reported_seq":4059,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.330542+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:46.330542+0000","last_peered":"2024-09-15T03:07:46.330542+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:46.330542+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3997,"log_dups_size":0,"ondisk_log_size":3997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32708210,"num_objects":3993,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3995,"num_write_kb":31956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1995}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4242","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:47.107276+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:47.107276+0000","last_peered":"2024-09-15T03:07:47.107276+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:47.107276+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34724665,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":33918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2137}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3984","reported_seq":4047,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.397457+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:46.397457+0000","last_peered":"2024-09-15T03:07:46.397457+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:46.397457+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32601714,"num_objects":3980,"num_object_clones":0,"num_object_copies":15920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":31845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4201","reported_seq":4257,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:45.999952+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:45.999952+0000","last_peered":"2024-09-15T03:07:45.999952+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:45.999952+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34387570,"num_objects":4198,"num_object_clones":0,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":33589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4199","reported_seq":4242,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.167915+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:46.167915+0000","last_peered":"2024-09-15T03:07:46.167915+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:46.167915+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34398208,"num_objects":4199,"num_object_clones":0,"num_object_copies":16796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":33592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2143}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4186","reported_seq":4251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.373357+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:46.373357+0000","last_peered":"2024-09-15T03:07:46.373357+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:46.373357+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34240114,"num_objects":4180,"num_object_clones":0,"num_object_copies":16720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":33452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2128}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4212","reported_seq":4275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.331816+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:46.331816+0000","last_peered":"2024-09-15T03:07:46.331816+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:46.331816+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4212,"log_dups_size":0,"ondisk_log_size":4212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34461298,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4135","reported_seq":4190,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.118426+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:46.118426+0000","last_peered":"2024-09-15T03:07:46.118426+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:46.118426+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4135,"log_dups_size":0,"ondisk_log_size":4135,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33803492,"num_objects":4127,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":33033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2085}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4164","reported_seq":4247,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.892475+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:46.892475+0000","last_peered":"2024-09-15T03:07:46.892475+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:46.892475+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34092658,"num_objects":4162,"num_object_clones":0,"num_object_copies":16648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4162,"num_write_kb":33294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2104}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4182","reported_seq":4240,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.053142+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:46.053142+0000","last_peered":"2024-09-15T03:07:46.053142+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:46.053142+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34214315,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":33420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4105","reported_seq":5662,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.103102+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:46.103102+0000","last_peered":"2024-09-15T03:07:46.103102+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:46.103102+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33618745,"num_objects":4104,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4104,"num_write_kb":32831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4175","reported_seq":4252,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.822315+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:46.822315+0000","last_peered":"2024-09-15T03:07:46.822315+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:46.822315+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34192185,"num_objects":4174,"num_object_clones":0,"num_object_copies":16696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4174,"num_write_kb":33391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2135}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4085","reported_seq":4135,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.021596+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:46.021596+0000","last_peered":"2024-09-15T03:07:46.021596+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:46.021596+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33429106,"num_objects":4081,"num_object_clones":0,"num_object_copies":16324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":32660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2045}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4149","reported_seq":4230,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.962329+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:46.962329+0000","last_peered":"2024-09-15T03:07:46.962329+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:46.962329+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33961586,"num_objects":4146,"num_object_clones":0,"num_object_copies":16584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":33173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2158}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4191","reported_seq":4252,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:46.434178+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:46.434178+0000","last_peered":"2024-09-15T03:07:46.434178+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:46.434178+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34289266,"num_objects":4186,"num_object_clones":0,"num_object_copies":16744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4189,"num_write_kb":33500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":542931516,"num_objects":66280,"num_object_clones":0,"num_object_copies":265120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66307,"num_write_kb":530338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":6225920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":955846656,"data_stored":955846656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66334,"ondisk_log_size":66334,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570768,"kb_used_data":264584,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93801072,"statfs":{"total":96636764160,"available":96052297728,"internally_reserved":0,"allocated":270934016,"data_stored":270775038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,0,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.355},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.55900000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.44900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.48199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567220,"kb_used_data":265708,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93804620,"statfs":{"total":96636764160,"available":96055930880,"internally_reserved":0,"allocated":272084992,"data_stored":271929160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,0,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.29899999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.502}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.33600000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":448324,"kb_used_data":137980,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93923516,"statfs":{"total":96636764160,"available":96177680384,"internally_reserved":0,"allocated":141291520,"data_stored":141132542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.222},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.47299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.253}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567080,"kb_used_data":265120,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93804760,"statfs":{"total":96636764160,"available":96056074240,"internally_reserved":0,"allocated":271482880,"data_stored":271323902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,3,5,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.57599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271392768,"data_stored":271392768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141262848,"data_stored":141262848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":272179200,"data_stored":272179200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271011840,"data_stored":271011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:52.382 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 136 20 67968 67948 3.90222 2.0625 0.190605 0.0400241 2024-09-15T03:07:53.384 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 137 20 68267 68247 3.89079 2.33594 0.0352264 0.0401372 2024-09-15T03:07:54.382 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 138 20 68568 68548 3.87964 2.35156 0.0616641 0.0402543 2024-09-15T03:07:54.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:55.382 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 139 20 68865 68845 3.86842 2.32031 0.14737 0.0403734 2024-09-15T03:07:56.383 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:07:56.382389+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0404996 2024-09-15T03:07:56.383 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:07:56.383 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 140 20 69145 69125 3.85641 2.1875 0.0777559 0.0404996 2024-09-15T03:07:56.437 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:56.451 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:56.484 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-15T03:07:54.688467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":554965564,"num_objects":67749,"num_object_clones":0,"num_object_copies":270996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67776,"num_write_kb":542090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":825,"num_bytes_recovered":6537216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67803,"ondisk_log_size":67803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2181336,"kb_used_data":957172,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375306024,"statfs":{"total":386547056640,"available":384313368576,"internally_reserved":0,"allocated":980144128,"data_stored":979511362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,17,8,11,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24403968,"num_objects":2979,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2979,"num_write_kb":23832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":589824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005392"},"pg_stats":[{"pgid":"1.f","version":"31'4223","reported_seq":4265,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.096044+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:51.096044+0000","last_peered":"2024-09-15T03:07:51.096044+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:51.096044+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34594816,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":33784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2247}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4096","reported_seq":4158,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.426462+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:51.426462+0000","last_peered":"2024-09-15T03:07:51.426462+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:51.426462+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4096,"log_dups_size":0,"ondisk_log_size":4096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33519218,"num_objects":4092,"num_object_clones":0,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4094,"num_write_kb":32748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2094}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4332","reported_seq":4413,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:52.119681+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:52.119681+0000","last_peered":"2024-09-15T03:07:52.119681+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:52.119681+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35461945,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4331,"num_write_kb":34638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4095","reported_seq":4157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.451827+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:51.451827+0000","last_peered":"2024-09-15T03:07:51.451827+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:51.451827+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4095,"log_dups_size":0,"ondisk_log_size":4095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33511026,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4093,"num_write_kb":32733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4286","reported_seq":4342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.138559+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:51.138559+0000","last_peered":"2024-09-15T03:07:51.138559+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:51.138559+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4286,"log_dups_size":0,"ondisk_log_size":4286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35083890,"num_objects":4283,"num_object_clones":0,"num_object_copies":17132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4284,"num_write_kb":34269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4278","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:50.938537+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:50.938537+0000","last_peered":"2024-09-15T03:07:50.938537+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:50.938537+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35045376,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4278,"num_write_kb":34224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2222}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4272","reported_seq":4335,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.334479+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:51.334479+0000","last_peered":"2024-09-15T03:07:51.334479+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:51.334479+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944626,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4270,"num_write_kb":34140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2214}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4301","reported_seq":4364,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.432512+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:51.432512+0000","last_peered":"2024-09-15T03:07:51.432512+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:51.432512+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4301,"log_dups_size":0,"ondisk_log_size":4301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35190386,"num_objects":4296,"num_object_clones":0,"num_object_copies":17184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4299,"num_write_kb":34380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2215}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4239","reported_seq":4295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.167451+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:51.167451+0000","last_peered":"2024-09-15T03:07:51.167451+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:51.167451+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34655460,"num_objects":4231,"num_object_clones":0,"num_object_copies":16924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":33865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4247","reported_seq":4330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:52.052679+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:52.052679+0000","last_peered":"2024-09-15T03:07:52.052679+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:52.052679+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34772594,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4245,"num_write_kb":33958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4271","reported_seq":4328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.023986+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:51.023986+0000","last_peered":"2024-09-15T03:07:51.023986+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:51.023986+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4271,"log_dups_size":0,"ondisk_log_size":4271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34943403,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4268,"num_write_kb":34132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2188}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4205","reported_seq":5839,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.174327+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:51.174327+0000","last_peered":"2024-09-15T03:07:51.174327+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:51.174327+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34437945,"num_objects":4204,"num_object_clones":0,"num_object_copies":16816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":33631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":799,"num_bytes_recovered":6537216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3010}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4282","reported_seq":4359,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.929691+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:51.929691+0000","last_peered":"2024-09-15T03:07:51.929691+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:51.929691+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35068729,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4281,"num_write_kb":34247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4165","reported_seq":4215,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:50.968950+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:50.968950+0000","last_peered":"2024-09-15T03:07:50.968950+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:50.968950+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4165,"log_dups_size":0,"ondisk_log_size":4165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34084466,"num_objects":4161,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4163,"num_write_kb":33300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4240","reported_seq":4321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.940086+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:51.940086+0000","last_peered":"2024-09-15T03:07:51.940086+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:51.940086+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34707058,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4238,"num_write_kb":33901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4271","reported_seq":4334,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.331785+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:51.331785+0000","last_peered":"2024-09-15T03:07:51.331785+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:51.331785+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4271,"log_dups_size":0,"ondisk_log_size":4271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34944626,"num_objects":4266,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4269,"num_write_kb":34140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":554965564,"num_objects":67749,"num_object_clones":0,"num_object_copies":270996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31858,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67776,"num_write_kb":542090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":825,"num_bytes_recovered":6537216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":980025344,"data_stored":980025344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67803,"ondisk_log_size":67803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580872,"kb_used_data":270524,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93790968,"statfs":{"total":96636764160,"available":96041951232,"internally_reserved":0,"allocated":277016576,"data_stored":276857598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.253}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":12.231999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577264,"kb_used_data":271592,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93794576,"statfs":{"total":96636764160,"available":96045645824,"internally_reserved":0,"allocated":278110208,"data_stored":277954376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":1.802},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.72899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.41299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":1.8759999999999999},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.69599999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446048,"kb_used_data":144024,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93925792,"statfs":{"total":96636764160,"available":96180011008,"internally_reserved":0,"allocated":147480576,"data_stored":147321598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.247},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":2.3399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.27600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.42499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577152,"kb_used_data":271032,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93794688,"statfs":{"total":96636764160,"available":96045760512,"internally_reserved":0,"allocated":277536768,"data_stored":277377790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,9,3,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.40400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":0.436},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":2.2210000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":2.1819999999999999},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":2.2599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277426176,"data_stored":277426176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147431424,"data_stored":147431424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278089728,"data_stored":278089728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":277078016,"data_stored":277078016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:56.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:57.383 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 141 20 69440 69420 3.84541 2.30469 0.0538939 0.0406214 2024-09-15T03:07:57.584 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:07:57.632 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-15T03:07:56.688967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":563739196,"num_objects":68820,"num_object_clones":0,"num_object_copies":275280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68847,"num_write_kb":550658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":6922240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68874,"ondisk_log_size":68874,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2184596,"kb_used_data":968756,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375302764,"statfs":{"total":386547056640,"available":384310030336,"internally_reserved":0,"allocated":992006144,"data_stored":991373378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,9,9,10,12,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33177600,"num_objects":4050,"num_object_clones":0,"num_object_copies":16200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4050,"num_write_kb":32400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005285"},"pg_stats":[{"pgid":"1.f","version":"31'4304","reported_seq":4346,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.059048+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:56.059048+0000","last_peered":"2024-09-15T03:07:56.059048+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:56.059048+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":34432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4183","reported_seq":4245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.460366+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:56.460366+0000","last_peered":"2024-09-15T03:07:56.460366+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:56.460366+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34231922,"num_objects":4179,"num_object_clones":0,"num_object_copies":16716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4181,"num_write_kb":33444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2181}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4332","reported_seq":4413,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:52.119681+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:52.119681+0000","last_peered":"2024-09-15T03:07:52.119681+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:52.119681+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35461945,"num_objects":4329,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4331,"num_write_kb":34638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4197","reported_seq":4260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.440274+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:56.440274+0000","last_peered":"2024-09-15T03:07:56.440274+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:56.440274+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34346610,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4374","reported_seq":4430,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.212890+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:56.212890+0000","last_peered":"2024-09-15T03:07:56.212890+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:56.212890+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804786,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4372,"num_write_kb":34973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4374","reported_seq":4418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.192656+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:56.192656+0000","last_peered":"2024-09-15T03:07:56.192656+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:56.192656+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35831808,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":34992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2318}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4352","reported_seq":4417,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.452277+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:56.452277+0000","last_peered":"2024-09-15T03:07:56.452277+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:56.452277+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599986,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4350,"num_write_kb":34780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2294}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4393","reported_seq":4456,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.406042+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:56.406042+0000","last_peered":"2024-09-15T03:07:56.406042+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:56.406042+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35944050,"num_objects":4388,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":35116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4327","reported_seq":4384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.178007+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:56.178007+0000","last_peered":"2024-09-15T03:07:56.178007+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:56.178007+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4327,"log_dups_size":0,"ondisk_log_size":4327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35376356,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":34569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4247","reported_seq":4330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:52.052679+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:52.052679+0000","last_peered":"2024-09-15T03:07:52.052679+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:52.052679+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34772594,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4245,"num_write_kb":33958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4345","reported_seq":4403,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.031719+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:56.031719+0000","last_peered":"2024-09-15T03:07:56.031719+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:56.031719+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35549611,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4342,"num_write_kb":34724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4300","reported_seq":6028,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.142585+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:56.142585+0000","last_peered":"2024-09-15T03:07:56.142585+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:56.142585+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35216185,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4299,"num_write_kb":34391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":846,"num_bytes_recovered":6922240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3152}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4282","reported_seq":4359,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.929691+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:51.929691+0000","last_peered":"2024-09-15T03:07:51.929691+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:51.929691+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35068729,"num_objects":4281,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4281,"num_write_kb":34247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4253","reported_seq":4303,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.056879+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:56.056879+0000","last_peered":"2024-09-15T03:07:56.056879+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:56.056879+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4253,"log_dups_size":0,"ondisk_log_size":4253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34805362,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4251,"num_write_kb":34004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2213}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4240","reported_seq":4321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:51.940086+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:51.940086+0000","last_peered":"2024-09-15T03:07:51.940086+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:51.940086+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34707058,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4238,"num_write_kb":33901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4371","reported_seq":4434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.474063+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:56.474063+0000","last_peered":"2024-09-15T03:07:56.474063+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:56.474063+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35763826,"num_objects":4366,"num_object_clones":0,"num_object_copies":17464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4369,"num_write_kb":34940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":563739196,"num_objects":68820,"num_object_clones":0,"num_object_copies":275280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68847,"num_write_kb":550658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":6922240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991895552,"data_stored":991895552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68874,"ondisk_log_size":68874,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574140,"kb_used_data":276276,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93797700,"statfs":{"total":96636764160,"available":96048844800,"internally_reserved":0,"allocated":282906624,"data_stored":282747646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,3,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.518}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.503},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.61799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577264,"kb_used_data":271592,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93794576,"statfs":{"total":96636764160,"available":96045645824,"internally_reserved":0,"allocated":278110208,"data_stored":277954376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":1.802},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.72899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.41299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":1.8759999999999999},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.69599999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446048,"kb_used_data":144024,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93925792,"statfs":{"total":96636764160,"available":96180011008,"internally_reserved":0,"allocated":147480576,"data_stored":147321598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.247},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":2.3399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.27600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.42499999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587144,"kb_used_data":276864,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93784696,"statfs":{"total":96636764160,"available":96035528704,"internally_reserved":0,"allocated":283508736,"data_stored":283349758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":4.1619999999999999},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":1.1479999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.375},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283336704,"data_stored":283336704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147431424,"data_stored":147431424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":278089728,"data_stored":278089728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283037696,"data_stored":283037696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:07:57.633 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4304", 'reported_seq': 4346, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.059048+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:07:56.059048+0000', 'last_peered': '2024-09-15T03:07:56.059048+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:07:56.059048+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4304, 'log_dups_size': 0, 'ondisk_log_size': 4304, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35258368, 'num_objects': 4304, 'num_object_clones': 0, 'num_object_copies': 17216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4304, 'num_write_kb': 34432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4183", 'reported_seq': 4245, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.460366+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:07:56.460366+0000', 'last_peered': '2024-09-15T03:07:56.460366+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:07:56.460366+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4183, 'log_dups_size': 0, 'ondisk_log_size': 4183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34231922, 'num_objects': 4179, 'num_object_clones': 0, 'num_object_copies': 16716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4179, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4181, 'num_write_kb': 33444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2181}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4332", 'reported_seq': 4413, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:52.119681+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:07:52.119681+0000', 'last_peered': '2024-09-15T03:07:52.119681+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:07:52.119681+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4332, 'log_dups_size': 0, 'ondisk_log_size': 4332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35461945, 'num_objects': 4329, 'num_object_clones': 0, 'num_object_copies': 17316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4331, 'num_write_kb': 34638, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2227}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4197", 'reported_seq': 4260, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.440274+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:07:56.440274+0000', 'last_peered': '2024-09-15T03:07:56.440274+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:07:56.440274+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4197, 'log_dups_size': 0, 'ondisk_log_size': 4197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34346610, 'num_objects': 4193, 'num_object_clones': 0, 'num_object_copies': 16772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4195, 'num_write_kb': 33549, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:07:57.634 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4374", 'reported_seq': 4430, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.212890+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:07:56.212890+0000', 'last_peered': '2024-09-15T03:07:56.212890+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:07:56.212890+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4374, 'log_dups_size': 0, 'ondisk_log_size': 4374, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35804786, 'num_objects': 4371, 'num_object_clones': 0, 'num_object_copies': 17484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4372, 'num_write_kb': 34973, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2264}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4374", 'reported_seq': 4418, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.192656+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:07:56.192656+0000', 'last_peered': '2024-09-15T03:07:56.192656+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:07:56.192656+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4374, 'log_dups_size': 0, 'ondisk_log_size': 4374, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35831808, 'num_objects': 4374, 'num_object_clones': 0, 'num_object_copies': 17496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4374, 'num_write_kb': 34992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2318}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4352", 'reported_seq': 4417, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.452277+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:07:56.452277+0000', 'last_peered': '2024-09-15T03:07:56.452277+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:07:56.452277+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4352, 'log_dups_size': 0, 'ondisk_log_size': 4352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35599986, 'num_objects': 4346, 'num_object_clones': 0, 'num_object_copies': 17384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4350, 'num_write_kb': 34780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2294}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4393", 'reported_seq': 4456, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.406042+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:07:56.406042+0000', 'last_peered': '2024-09-15T03:07:56.406042+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:07:56.406042+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4393, 'log_dups_size': 0, 'ondisk_log_size': 4393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35944050, 'num_objects': 4388, 'num_object_clones': 0, 'num_object_copies': 17552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4391, 'num_write_kb': 35116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4327", 'reported_seq': 4384, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.178007+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:07:56.178007+0000', 'last_peered': '2024-09-15T03:07:56.178007+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:07:56.178007+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4327, 'log_dups_size': 0, 'ondisk_log_size': 4327, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35376356, 'num_objects': 4319, 'num_object_clones': 0, 'num_object_copies': 17276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4324, 'num_write_kb': 34569, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2277}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4247", 'reported_seq': 4330, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:52.052679+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:07:52.052679+0000', 'last_peered': '2024-09-15T03:07:52.052679+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:07:52.052679+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4247, 'log_dups_size': 0, 'ondisk_log_size': 4247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34772594, 'num_objects': 4245, 'num_object_clones': 0, 'num_object_copies': 16980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4245, 'num_write_kb': 33958, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:57.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4345", 'reported_seq': 4403, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.031719+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:07:56.031719+0000', 'last_peered': '2024-09-15T03:07:56.031719+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:07:56.031719+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4345, 'log_dups_size': 0, 'ondisk_log_size': 4345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35549611, 'num_objects': 4340, 'num_object_clones': 0, 'num_object_copies': 17360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4342, 'num_write_kb': 34724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2262}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4300", 'reported_seq': 6028, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.142585+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:07:56.142585+0000', 'last_peered': '2024-09-15T03:07:56.142585+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:07:56.142585+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4300, 'log_dups_size': 0, 'ondisk_log_size': 4300, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35216185, 'num_objects': 4299, 'num_object_clones': 0, 'num_object_copies': 17196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1147, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4299, 'num_write_kb': 34391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 846, 'num_bytes_recovered': 6922240, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3152}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4282", 'reported_seq': 4359, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:51.929691+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:07:51.929691+0000', 'last_peered': '2024-09-15T03:07:51.929691+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:07:51.929691+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4282, 'log_dups_size': 0, 'ondisk_log_size': 4282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35068729, 'num_objects': 4281, 'num_object_clones': 0, 'num_object_copies': 17124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4281, 'num_write_kb': 34247, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4253", 'reported_seq': 4303, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.056879+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:07:56.056879+0000', 'last_peered': '2024-09-15T03:07:56.056879+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:07:56.056879+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4253, 'log_dups_size': 0, 'ondisk_log_size': 4253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34805362, 'num_objects': 4249, 'num_object_clones': 0, 'num_object_copies': 16996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4251, 'num_write_kb': 34004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2213}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4240", 'reported_seq': 4321, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:51.940086+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:07:51.940086+0000', 'last_peered': '2024-09-15T03:07:51.940086+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:07:51.940086+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4240, 'log_dups_size': 0, 'ondisk_log_size': 4240, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34707058, 'num_objects': 4237, 'num_object_clones': 0, 'num_object_copies': 16948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4238, 'num_write_kb': 33901, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2249}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:07:57.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:07:57.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4371", 'reported_seq': 4434, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:07:56.474063+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:07:56.474063+0000', 'last_peered': '2024-09-15T03:07:56.474063+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:07:56.474063+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4371, 'log_dups_size': 0, 'ondisk_log_size': 4371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35763826, 'num_objects': 4366, 'num_object_clones': 0, 'num_object_copies': 17464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4369, 'num_write_kb': 34940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2284}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:07:57.637 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:07:58.383 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 142 20 69716 69696 3.83352 2.15625 0.149917 0.0407417 2024-09-15T03:07:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:07:59.263 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31811/276812 objects degraded (11.492%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":202,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":69203,"data_bytes":566876732,"bytes_used":2261729280,"bytes_avail":384285327360,"bytes_total":386547056640,"degraded_objects":31811,"degraded_total":276812,"degraded_ratio":0.11491915090386255,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":58000,"recovering_keys_per_sec":0,"num_objects_recovered":85,"num_bytes_recovered":696320,"num_keys_recovered":0,"write_bytes_sec":2242922,"read_op_per_sec":0,"write_op_per_sec":273},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:07:59.349 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:07:59.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:07:59.384 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 143 20 69995 69975 3.82195 2.17969 0.153502 0.0408588 2024-09-15T03:08:00.383 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 144 20 70284 70264 3.81109 2.25781 0.0337906 0.0409843 2024-09-15T03:08:00.780 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:00.842 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:00.884 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-15T03:08:00.690049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":566876732,"num_objects":69203,"num_object_clones":0,"num_object_copies":276812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69230,"num_write_kb":553722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":6922240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69257,"ondisk_log_size":69257,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2208720,"kb_used_data":980400,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375278640,"statfs":{"total":386547056640,"available":384285327360,"internally_reserved":0,"allocated":1003929600,"data_stored":1003296834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,7,6,7,11,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23945216,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":23384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":696320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005369"},"pg_stats":[{"pgid":"1.f","version":"31'4304","reported_seq":4346,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.059048+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:07:56.059048+0000","last_peered":"2024-09-15T03:07:56.059048+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:07:56.059048+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4304,"num_write_kb":34432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4183","reported_seq":4245,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.460366+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:07:56.460366+0000","last_peered":"2024-09-15T03:07:56.460366+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:07:56.460366+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34231922,"num_objects":4179,"num_object_clones":0,"num_object_copies":16716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4181,"num_write_kb":33444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2181}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4420","reported_seq":4501,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:57.049292+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:07:57.049292+0000","last_peered":"2024-09-15T03:07:57.049292+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:07:57.049292+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36182841,"num_objects":4417,"num_object_clones":0,"num_object_copies":17668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4419,"num_write_kb":35342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2315}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4197","reported_seq":4260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.440274+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:07:56.440274+0000","last_peered":"2024-09-15T03:07:56.440274+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:07:56.440274+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34346610,"num_objects":4193,"num_object_clones":0,"num_object_copies":16772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":33549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4374","reported_seq":4430,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.212890+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:07:56.212890+0000","last_peered":"2024-09-15T03:07:56.212890+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:07:56.212890+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804786,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4372,"num_write_kb":34973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4374","reported_seq":4418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.192656+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:07:56.192656+0000","last_peered":"2024-09-15T03:07:56.192656+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:07:56.192656+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35831808,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":34992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2318}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4352","reported_seq":4417,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.452277+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:07:56.452277+0000","last_peered":"2024-09-15T03:07:56.452277+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:07:56.452277+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4352,"log_dups_size":0,"ondisk_log_size":4352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599986,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4350,"num_write_kb":34780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2294}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4393","reported_seq":4456,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.406042+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:07:56.406042+0000","last_peered":"2024-09-15T03:07:56.406042+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:07:56.406042+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35944050,"num_objects":4388,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":35116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4327","reported_seq":4384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.178007+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:07:56.178007+0000","last_peered":"2024-09-15T03:07:56.178007+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:07:56.178007+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4327,"log_dups_size":0,"ondisk_log_size":4327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35376356,"num_objects":4319,"num_object_clones":0,"num_object_copies":17276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":34569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4338","reported_seq":4421,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:57.132057+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:07:57.132057+0000","last_peered":"2024-09-15T03:07:57.132057+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:07:57.132057+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35518066,"num_objects":4336,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4336,"num_write_kb":34686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2278}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4345","reported_seq":4403,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.031719+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:07:56.031719+0000","last_peered":"2024-09-15T03:07:56.031719+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:07:56.031719+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35549611,"num_objects":4340,"num_object_clones":0,"num_object_copies":17360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4342,"num_write_kb":34724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4300","reported_seq":6028,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.142585+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:07:56.142585+0000","last_peered":"2024-09-15T03:07:56.142585+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:07:56.142585+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35216185,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4299,"num_write_kb":34391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":846,"num_bytes_recovered":6922240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3152}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4377","reported_seq":4454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:57.029792+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:07:57.029792+0000","last_peered":"2024-09-15T03:07:57.029792+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:07:57.029792+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4377,"log_dups_size":0,"ondisk_log_size":4377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35846969,"num_objects":4376,"num_object_clones":0,"num_object_copies":17504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4376,"num_write_kb":35007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2337}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4253","reported_seq":4303,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.056879+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:07:56.056879+0000","last_peered":"2024-09-15T03:07:56.056879+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:07:56.056879+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4253,"log_dups_size":0,"ondisk_log_size":4253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34805362,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4251,"num_write_kb":34004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2213}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4349","reported_seq":4427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:57.139188+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:07:57.139188+0000","last_peered":"2024-09-15T03:07:57.139188+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:07:57.139188+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35599986,"num_objects":4346,"num_object_clones":0,"num_object_copies":17384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4347,"num_write_kb":34773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4371","reported_seq":4434,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:07:56.474063+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:07:56.474063+0000","last_peered":"2024-09-15T03:07:56.474063+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:07:56.474063+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4371,"log_dups_size":0,"ondisk_log_size":4371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35763826,"num_objects":4366,"num_object_clones":0,"num_object_copies":17464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4369,"num_write_kb":34940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2284}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":566876732,"num_objects":69203,"num_object_clones":0,"num_object_copies":276812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69230,"num_write_kb":553722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":6922240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1004056576,"data_stored":1004056576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69257,"ondisk_log_size":69257,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574140,"kb_used_data":276276,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93797700,"statfs":{"total":96636764160,"available":96048844800,"internally_reserved":0,"allocated":282906624,"data_stored":282747646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,3,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.518}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.503},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.61799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591240,"kb_used_data":277248,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93780600,"statfs":{"total":96636764160,"available":96031334400,"internally_reserved":0,"allocated":283901952,"data_stored":283746120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":1.802},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.72899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.41299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":1.8759999999999999},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.69599999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":456196,"kb_used_data":150012,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93915644,"statfs":{"total":96636764160,"available":96169619456,"internally_reserved":0,"allocated":153612288,"data_stored":153453310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.224}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":4.7160000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":0.251}]}]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587144,"kb_used_data":276864,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93784696,"statfs":{"total":96636764160,"available":96035528704,"internally_reserved":0,"allocated":283508736,"data_stored":283349758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":4.1619999999999999},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":1.1479999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.375},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":283336704,"data_stored":283336704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":153567232,"data_stored":153567232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284114944,"data_stored":284114944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":283037696,"data_stored":283037696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:01.383 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 145 20 70574 70554 3.80043 2.26562 0.0616947 0.0410964 2024-09-15T03:08:02.385 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 146 20 70871 70851 3.79024 2.32031 0.0644601 0.041207 2024-09-15T03:08:03.385 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 147 20 71158 71138 3.77971 2.24219 0.0282787 0.0413209 2024-09-15T03:08:03.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:04.385 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 148 20 71447 71427 3.76943 2.25781 0.048301 0.0414306 2024-09-15T03:08:05.126 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:05.144 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:05.157 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-15T03:08:04.691110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":578607676,"num_objects":70635,"num_object_clones":0,"num_object_copies":282540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70662,"num_write_kb":565178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":7241728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70689,"ondisk_log_size":70689,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2227404,"kb_used_data":1003244,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375259956,"statfs":{"total":386547056640,"available":384266194944,"internally_reserved":0,"allocated":1027321856,"data_stored":1026689090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,5,9,8,7,11,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23642112,"num_objects":2886,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":23088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003215"},"pg_stats":[{"pgid":"1.f","version":"31'4397","reported_seq":4439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.211625+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:01.211625+0000","last_peered":"2024-09-15T03:08:01.211625+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:01.211625+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36020224,"num_objects":4397,"num_object_clones":0,"num_object_copies":17588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4397,"num_write_kb":35176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4259","reported_seq":4321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.187546+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:01.187546+0000","last_peered":"2024-09-15T03:08:01.187546+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:01.187546+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34854514,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":34052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4503","reported_seq":4584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:02.116762+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:02.116762+0000","last_peered":"2024-09-15T03:08:02.116762+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:02.116762+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36862777,"num_objects":4500,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":36006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4298","reported_seq":4361,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.373385+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:01.373385+0000","last_peered":"2024-09-15T03:08:01.373385+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:01.373385+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35174002,"num_objects":4294,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4296,"num_write_kb":34357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4466","reported_seq":4522,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.228809+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:01.228809+0000","last_peered":"2024-09-15T03:08:01.228809+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:01.228809+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36558450,"num_objects":4463,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4470","reported_seq":4511,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.226565+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:01.226565+0000","last_peered":"2024-09-15T03:08:01.226565+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:01.226565+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36618240,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4470,"num_write_kb":35760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4450","reported_seq":4515,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.416842+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:01.416842+0000","last_peered":"2024-09-15T03:08:01.416842+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:01.416842+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4450,"log_dups_size":0,"ondisk_log_size":4450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402802,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":35564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2392}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4485","reported_seq":4548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.394422+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:01.394422+0000","last_peered":"2024-09-15T03:08:01.394422+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:01.394422+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697714,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4401","reported_seq":4458,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:00.893050+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:00.893050+0000","last_peered":"2024-09-15T03:08:00.893050+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:00.893050+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35982564,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4398,"num_write_kb":35161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4420","reported_seq":4503,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:02.105462+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:02.105462+0000","last_peered":"2024-09-15T03:08:02.105462+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:02.105462+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189810,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4441","reported_seq":4499,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.093938+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:01.093938+0000","last_peered":"2024-09-15T03:08:01.093938+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:01.093938+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36336043,"num_objects":4436,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4438,"num_write_kb":35492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4390","reported_seq":6196,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.211652+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:01.211652+0000","last_peered":"2024-09-15T03:08:01.211652+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:01.211652+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35953465,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":7241728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4465","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:02.042863+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:02.042863+0000","last_peered":"2024-09-15T03:08:02.042863+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:02.042863+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4465,"log_dups_size":0,"ondisk_log_size":4465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36567865,"num_objects":4464,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2425}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4342","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.145236+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:01.145236+0000","last_peered":"2024-09-15T03:08:01.145236+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:01.145236+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35534450,"num_objects":4338,"num_object_clones":0,"num_object_copies":17352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":34716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2302}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4440","reported_seq":4521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:02.089543+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:02.089543+0000","last_peered":"2024-09-15T03:08:02.089543+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:02.089543+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36345458,"num_objects":4437,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4438,"num_write_kb":35501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4462","reported_seq":4524,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.446635+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:01.446635+0000","last_peered":"2024-09-15T03:08:01.446635+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:01.446635+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36509298,"num_objects":4457,"num_object_clones":0,"num_object_copies":17828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":35668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":578607676,"num_objects":70635,"num_object_clones":0,"num_object_copies":282540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70662,"num_write_kb":565178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":7241728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1027825664,"data_stored":1027825664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70689,"ondisk_log_size":70689,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588108,"kb_used_data":281924,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93783732,"statfs":{"total":96636764160,"available":96034541568,"internally_reserved":0,"allocated":288690176,"data_stored":288531198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,5,8,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.253},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":2.7200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":13.176},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":13.510999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":2.577}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584460,"kb_used_data":282948,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93787380,"statfs":{"total":96636764160,"available":96038277120,"internally_reserved":0,"allocated":289738752,"data_stored":289582920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":4.1520000000000001},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.22}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.253},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.55700000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.27800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470364,"kb_used_data":155860,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93901476,"statfs":{"total":96636764160,"available":96155111424,"internally_reserved":0,"allocated":159600640,"data_stored":159441662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":4.4649999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":2.5720000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.26200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.246},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":2.6400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584472,"kb_used_data":282512,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93787368,"statfs":{"total":96636764160,"available":96038264832,"internally_reserved":0,"allocated":289292288,"data_stored":289133310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":0.753},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":3.0550000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.378}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289214464,"data_stored":289214464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159670272,"data_stored":159670272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290029568,"data_stored":290029568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":288911360,"data_stored":288911360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:05.159 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:05.385 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 149 20 71736 71716 3.75928 2.25781 0.0178211 0.0415451 2024-09-15T03:08:06.387 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 150 20 72020 72000 3.74897 2.21875 0.0391224 0.0416588 2024-09-15T03:08:06.543 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:06.589 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-15T03:08:04.691110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":578607676,"num_objects":70635,"num_object_clones":0,"num_object_copies":282540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70662,"num_write_kb":565178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":7241728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70689,"ondisk_log_size":70689,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2227404,"kb_used_data":1003244,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375259956,"statfs":{"total":386547056640,"available":384266194944,"internally_reserved":0,"allocated":1027321856,"data_stored":1026689090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,5,9,8,7,11,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23642112,"num_objects":2886,"num_object_clones":0,"num_object_copies":11544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2886,"num_write_kb":23088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003215"},"pg_stats":[{"pgid":"1.f","version":"31'4397","reported_seq":4439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.211625+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:01.211625+0000","last_peered":"2024-09-15T03:08:01.211625+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:01.211625+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36020224,"num_objects":4397,"num_object_clones":0,"num_object_copies":17588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4397,"num_write_kb":35176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4259","reported_seq":4321,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.187546+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:01.187546+0000","last_peered":"2024-09-15T03:08:01.187546+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:01.187546+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34854514,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":34052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4503","reported_seq":4584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:02.116762+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:02.116762+0000","last_peered":"2024-09-15T03:08:02.116762+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:02.116762+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36862777,"num_objects":4500,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":36006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4298","reported_seq":4361,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.373385+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:01.373385+0000","last_peered":"2024-09-15T03:08:01.373385+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:01.373385+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35174002,"num_objects":4294,"num_object_clones":0,"num_object_copies":17176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4296,"num_write_kb":34357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4466","reported_seq":4522,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.228809+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:01.228809+0000","last_peered":"2024-09-15T03:08:01.228809+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:01.228809+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36558450,"num_objects":4463,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2356}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4470","reported_seq":4511,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.226565+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:01.226565+0000","last_peered":"2024-09-15T03:08:01.226565+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:01.226565+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4470,"log_dups_size":0,"ondisk_log_size":4470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36618240,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4470,"num_write_kb":35760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4450","reported_seq":4515,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.416842+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:01.416842+0000","last_peered":"2024-09-15T03:08:01.416842+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:01.416842+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4450,"log_dups_size":0,"ondisk_log_size":4450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402802,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":35564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2392}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4485","reported_seq":4548,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.394422+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:01.394422+0000","last_peered":"2024-09-15T03:08:01.394422+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:01.394422+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36697714,"num_objects":4480,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":35852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2399}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4401","reported_seq":4458,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:00.893050+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:00.893050+0000","last_peered":"2024-09-15T03:08:00.893050+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:00.893050+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35982564,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4398,"num_write_kb":35161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4420","reported_seq":4503,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:02.105462+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:02.105462+0000","last_peered":"2024-09-15T03:08:02.105462+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:02.105462+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189810,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":35342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4441","reported_seq":4499,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.093938+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:01.093938+0000","last_peered":"2024-09-15T03:08:01.093938+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:01.093938+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4441,"log_dups_size":0,"ondisk_log_size":4441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36336043,"num_objects":4436,"num_object_clones":0,"num_object_copies":17744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4438,"num_write_kb":35492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4390","reported_seq":6196,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.211652+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:01.211652+0000","last_peered":"2024-09-15T03:08:01.211652+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:01.211652+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4390,"log_dups_size":0,"ondisk_log_size":4390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35953465,"num_objects":4389,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":35111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":7241728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3281}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4465","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:02.042863+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:02.042863+0000","last_peered":"2024-09-15T03:08:02.042863+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:02.042863+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4465,"log_dups_size":0,"ondisk_log_size":4465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36567865,"num_objects":4464,"num_object_clones":0,"num_object_copies":17856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":35711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2425}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4342","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.145236+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:01.145236+0000","last_peered":"2024-09-15T03:08:01.145236+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:01.145236+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35534450,"num_objects":4338,"num_object_clones":0,"num_object_copies":17352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":34716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2302}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4440","reported_seq":4521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:02.089543+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:02.089543+0000","last_peered":"2024-09-15T03:08:02.089543+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:02.089543+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36345458,"num_objects":4437,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4438,"num_write_kb":35501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4462","reported_seq":4524,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:01.446635+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:01.446635+0000","last_peered":"2024-09-15T03:08:01.446635+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:01.446635+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36509298,"num_objects":4457,"num_object_clones":0,"num_object_copies":17828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":35668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":578607676,"num_objects":70635,"num_object_clones":0,"num_object_copies":282540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70662,"num_write_kb":565178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":911,"num_bytes_recovered":7241728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1027825664,"data_stored":1027825664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70689,"ondisk_log_size":70689,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588108,"kb_used_data":281924,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93783732,"statfs":{"total":96636764160,"available":96034541568,"internally_reserved":0,"allocated":288690176,"data_stored":288531198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,5,8,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.253},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":2.7200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":13.176},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.39700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":13.510999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":2.577}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584460,"kb_used_data":282948,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93787380,"statfs":{"total":96636764160,"available":96038277120,"internally_reserved":0,"allocated":289738752,"data_stored":289582920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":4.1520000000000001},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.22}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.253},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.55700000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.27800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470364,"kb_used_data":155860,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93901476,"statfs":{"total":96636764160,"available":96155111424,"internally_reserved":0,"allocated":159600640,"data_stored":159441662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":2.2669999999999999,"15min":2.2669999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":18.091000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":9.4339999999999993,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":4.4649999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":9.1620000000000008,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":2.5720000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.7410000000000001,"15min":1.7410000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":8.3859999999999992,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.26200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:07:06 2024","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.659,"15min":1.659},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":13.414999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.246},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":6.0090000000000003,"5min":6.0090000000000003,"15min":6.0090000000000003},"last":2.6400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584472,"kb_used_data":282512,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93787368,"statfs":{"total":96636764160,"available":96038264832,"internally_reserved":0,"allocated":289292288,"data_stored":289133310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":0.753},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":3.0550000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.378}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289214464,"data_stored":289214464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159670272,"data_stored":159670272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":290029568,"data_stored":290029568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":288911360,"data_stored":288911360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4397", 'reported_seq': 4439, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.211625+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:08:01.211625+0000', 'last_peered': '2024-09-15T03:08:01.211625+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:08:01.211625+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4397, 'log_dups_size': 0, 'ondisk_log_size': 4397, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36020224, 'num_objects': 4397, 'num_object_clones': 0, 'num_object_copies': 17588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4397, 'num_write_kb': 35176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2421}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4259", 'reported_seq': 4321, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.187546+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:08:01.187546+0000', 'last_peered': '2024-09-15T03:08:01.187546+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:08:01.187546+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4259, 'log_dups_size': 0, 'ondisk_log_size': 4259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34854514, 'num_objects': 4255, 'num_object_clones': 0, 'num_object_copies': 17020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4257, 'num_write_kb': 34052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4503", 'reported_seq': 4584, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:02.116762+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:08:02.116762+0000', 'last_peered': '2024-09-15T03:08:02.116762+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:08:02.116762+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36862777, 'num_objects': 4500, 'num_object_clones': 0, 'num_object_copies': 18000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4502, 'num_write_kb': 36006, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2398}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4298", 'reported_seq': 4361, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.373385+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:08:01.373385+0000', 'last_peered': '2024-09-15T03:08:01.373385+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:08:01.373385+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4298, 'log_dups_size': 0, 'ondisk_log_size': 4298, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35174002, 'num_objects': 4294, 'num_object_clones': 0, 'num_object_copies': 17176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4296, 'num_write_kb': 34357, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2351}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:06.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4466", 'reported_seq': 4522, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.228809+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:08:01.228809+0000', 'last_peered': '2024-09-15T03:08:01.228809+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:08:01.228809+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4466, 'log_dups_size': 0, 'ondisk_log_size': 4466, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36558450, 'num_objects': 4463, 'num_object_clones': 0, 'num_object_copies': 17852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4464, 'num_write_kb': 35709, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2356}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4470", 'reported_seq': 4511, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.226565+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:08:01.226565+0000', 'last_peered': '2024-09-15T03:08:01.226565+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:08:01.226565+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4470, 'log_dups_size': 0, 'ondisk_log_size': 4470, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36618240, 'num_objects': 4470, 'num_object_clones': 0, 'num_object_copies': 17880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4470, 'num_write_kb': 35760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2414}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4450", 'reported_seq': 4515, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.416842+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:08:01.416842+0000', 'last_peered': '2024-09-15T03:08:01.416842+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:08:01.416842+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4450, 'log_dups_size': 0, 'ondisk_log_size': 4450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36402802, 'num_objects': 4444, 'num_object_clones': 0, 'num_object_copies': 17776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4448, 'num_write_kb': 35564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2392}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4485", 'reported_seq': 4548, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.394422+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:08:01.394422+0000', 'last_peered': '2024-09-15T03:08:01.394422+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:08:01.394422+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4485, 'log_dups_size': 0, 'ondisk_log_size': 4485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36697714, 'num_objects': 4480, 'num_object_clones': 0, 'num_object_copies': 17920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4483, 'num_write_kb': 35852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2399}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4401", 'reported_seq': 4458, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:00.893050+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:08:00.893050+0000', 'last_peered': '2024-09-15T03:08:00.893050+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:08:00.893050+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4401, 'log_dups_size': 0, 'ondisk_log_size': 4401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35982564, 'num_objects': 4393, 'num_object_clones': 0, 'num_object_copies': 17572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4398, 'num_write_kb': 35161, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:06.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:08:06.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4420", 'reported_seq': 4503, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:02.105462+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:08:02.105462+0000', 'last_peered': '2024-09-15T03:08:02.105462+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:08:02.105462+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4420, 'log_dups_size': 0, 'ondisk_log_size': 4420, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36189810, 'num_objects': 4418, 'num_object_clones': 0, 'num_object_copies': 17672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4418, 'num_write_kb': 35342, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2360}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:06.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:08:06.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4441", 'reported_seq': 4499, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.093938+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:08:01.093938+0000', 'last_peered': '2024-09-15T03:08:01.093938+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:08:01.093938+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4441, 'log_dups_size': 0, 'ondisk_log_size': 4441, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36336043, 'num_objects': 4436, 'num_object_clones': 0, 'num_object_copies': 17744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4438, 'num_write_kb': 35492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2358}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:06.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:08:06.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4390", 'reported_seq': 6196, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.211652+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:08:01.211652+0000', 'last_peered': '2024-09-15T03:08:01.211652+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:08:01.211652+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4390, 'log_dups_size': 0, 'ondisk_log_size': 4390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35953465, 'num_objects': 4389, 'num_object_clones': 0, 'num_object_copies': 17556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1108, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4389, 'num_write_kb': 35111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 885, 'num_bytes_recovered': 7241728, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:06.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:08:06.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4465", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:02.042863+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:08:02.042863+0000', 'last_peered': '2024-09-15T03:08:02.042863+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:08:02.042863+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4465, 'log_dups_size': 0, 'ondisk_log_size': 4465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36567865, 'num_objects': 4464, 'num_object_clones': 0, 'num_object_copies': 17856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4464, 'num_write_kb': 35711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2425}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:06.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:08:06.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4342", 'reported_seq': 4391, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.145236+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:08:01.145236+0000', 'last_peered': '2024-09-15T03:08:01.145236+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:08:01.145236+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4342, 'log_dups_size': 0, 'ondisk_log_size': 4342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35534450, 'num_objects': 4338, 'num_object_clones': 0, 'num_object_copies': 17352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4340, 'num_write_kb': 34716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:06.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:08:06.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4440", 'reported_seq': 4521, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:02.089543+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:08:02.089543+0000', 'last_peered': '2024-09-15T03:08:02.089543+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:08:02.089543+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4440, 'log_dups_size': 0, 'ondisk_log_size': 4440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36345458, 'num_objects': 4437, 'num_object_clones': 0, 'num_object_copies': 17748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4438, 'num_write_kb': 35501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:06.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:08:06.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4462", 'reported_seq': 4524, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:01.446635+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:08:01.446635+0000', 'last_peered': '2024-09-15T03:08:01.446635+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:08:01.446635+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4462, 'log_dups_size': 0, 'ondisk_log_size': 4462, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36509298, 'num_objects': 4457, 'num_object_clones': 0, 'num_object_copies': 17828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4460, 'num_write_kb': 35668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:06.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:08:07.388 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 151 20 72297 72277 3.73844 2.16406 0.0425559 0.0417799 2024-09-15T03:08:08.390 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 152 20 72570 72550 3.72784 2.13281 0.076977 0.0419009 2024-09-15T03:08:08.496 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:08.502 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31747/287036 objects degraded (11.060%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":211,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":71759,"data_bytes":587815484,"bytes_used":2301263872,"bytes_avail":384245792768,"bytes_total":386547056640,"degraded_objects":31747,"degraded_total":287036,"degraded_ratio":0.11060285121030117,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":75756,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"write_bytes_sec":2736784,"read_op_per_sec":0,"write_op_per_sec":334},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:08:08.534 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:08:08.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:09.390 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 153 20 72845 72825 3.71752 2.14844 0.06975 0.0420133 2024-09-15T03:08:09.685 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:09.697 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:09.726 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-15T03:08:08.692108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590633532,"num_objects":72103,"num_object_clones":0,"num_object_copies":288412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72130,"num_write_kb":576922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":7446528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72157,"ondisk_log_size":72157,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2250644,"kb_used_data":1026484,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375236716,"statfs":{"total":386547056640,"available":384242397184,"internally_reserved":0,"allocated":1051119616,"data_stored":1050486850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,8,11,3,12,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26894336,"num_objects":3283,"num_object_clones":0,"num_object_copies":13132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-64,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3283,"num_write_kb":26264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003140"},"pg_stats":[{"pgid":"1.f","version":"31'4494","reported_seq":4535,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.283335+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:06.283335+0000","last_peered":"2024-09-15T03:08:06.283335+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:06.283335+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36814848,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4494,"num_write_kb":35952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2518}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4347","reported_seq":4409,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.378252+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:06.378252+0000","last_peered":"2024-09-15T03:08:06.378252+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:06.378252+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4347,"log_dups_size":0,"ondisk_log_size":4347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35575410,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":34756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2345}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4602","reported_seq":4682,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:07.098235+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:07.098235+0000","last_peered":"2024-09-15T03:08:07.098235+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:07.098235+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37673785,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4601,"num_write_kb":36798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2497}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4379","reported_seq":4442,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.485096+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:06.485096+0000","last_peered":"2024-09-15T03:08:06.485096+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:06.485096+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35837554,"num_objects":4375,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4377,"num_write_kb":35005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4545","reported_seq":4600,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.262639+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:06.262639+0000","last_peered":"2024-09-15T03:08:06.262639+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:06.262639+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4545,"log_dups_size":0,"ondisk_log_size":4545,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37205618,"num_objects":4542,"num_object_clones":0,"num_object_copies":18168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":36341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4573","reported_seq":4617,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.098720+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:06.098720+0000","last_peered":"2024-09-15T03:08:06.098720+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:06.098720+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37462016,"num_objects":4573,"num_object_clones":0,"num_object_copies":18292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4573,"num_write_kb":36584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4543","reported_seq":4607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.505995+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:06.505995+0000","last_peered":"2024-09-15T03:08:06.505995+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:06.505995+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37164658,"num_objects":4537,"num_object_clones":0,"num_object_copies":18148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4541,"num_write_kb":36308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4576","reported_seq":4639,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.528663+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:06.528663+0000","last_peered":"2024-09-15T03:08:06.528663+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:06.528663+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4576,"log_dups_size":0,"ondisk_log_size":4576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37443186,"num_objects":4571,"num_object_clones":0,"num_object_copies":18284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4574,"num_write_kb":36580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4504","reported_seq":4561,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.152904+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:06.152904+0000","last_peered":"2024-09-15T03:08:06.152904+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:06.152904+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36826340,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":35985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4510","reported_seq":4593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:07.155126+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:07.155126+0000","last_peered":"2024-09-15T03:08:07.155126+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:07.155126+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4510,"log_dups_size":0,"ondisk_log_size":4510,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36927090,"num_objects":4508,"num_object_clones":0,"num_object_copies":18032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4508,"num_write_kb":36062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2450}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4541","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.162369+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:06.162369+0000","last_peered":"2024-09-15T03:08:06.162369+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:06.162369+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37155243,"num_objects":4536,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4538,"num_write_kb":36292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4503","reported_seq":6359,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.211147+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:06.211147+0000","last_peered":"2024-09-15T03:08:06.211147+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:06.211147+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36879161,"num_objects":4502,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4502,"num_write_kb":36015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":910,"num_bytes_recovered":7446528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4555","reported_seq":4631,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:07.238444+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:07.238444+0000","last_peered":"2024-09-15T03:08:07.238444+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:07.238444+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37305145,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4554,"num_write_kb":36431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2515}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4423","reported_seq":4473,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.224762+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:06.224762+0000","last_peered":"2024-09-15T03:08:06.224762+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:06.224762+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4423,"log_dups_size":0,"ondisk_log_size":4423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36198002,"num_objects":4419,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4421,"num_write_kb":35364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2383}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4505","reported_seq":4586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:07.133326+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:07.133326+0000","last_peered":"2024-09-15T03:08:07.133326+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:07.133326+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36877938,"num_objects":4502,"num_object_clones":0,"num_object_copies":18008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":36021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4557","reported_seq":4620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:06.344632+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:06.344632+0000","last_peered":"2024-09-15T03:08:06.344632+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:06.344632+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4557,"log_dups_size":0,"ondisk_log_size":4557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37287538,"num_objects":4552,"num_object_clones":0,"num_object_copies":18208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4555,"num_write_kb":36428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2470}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":590633532,"num_objects":72103,"num_object_clones":0,"num_object_copies":288412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72130,"num_write_kb":576922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":7446528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1051660288,"data_stored":1051660288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72157,"ondisk_log_size":72157,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598052,"kb_used_data":287708,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93773788,"statfs":{"total":96636764160,"available":96024358912,"internally_reserved":0,"allocated":294612992,"data_stored":294454014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,0,5,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.246}]},{"osd":1,"last update":"Sun Sep 15 03:07:07 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.633,"15min":1.633},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.179,"5min":1.179,"15min":1.179},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":4.5369999999999999},"last":0.68100000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.65200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594484,"kb_used_data":288812,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93777356,"statfs":{"total":96636764160,"available":96028012544,"internally_reserved":0,"allocated":295743488,"data_stored":295587656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.182},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.26600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":0.318}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":8.2810000000000006},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":10.231}]}]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463656,"kb_used_data":161632,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93908184,"statfs":{"total":96636764160,"available":96161980416,"internally_reserved":0,"allocated":165511168,"data_stored":165352190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.27200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.36499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.315}]}]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":594452,"kb_used_data":288332,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93777388,"statfs":{"total":96636764160,"available":96028045312,"internally_reserved":0,"allocated":295251968,"data_stored":295092990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":6.6929999999999996,"5min":6.6929999999999996,"15min":6.6929999999999996},"last":1.615},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":10.417999999999999,"5min":10.417999999999999,"15min":10.417999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":2.0089999999999999},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":295153664,"data_stored":295153664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":165720064,"data_stored":165720064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295923712,"data_stored":295923712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294862848,"data_stored":294862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:10.391 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 154 20 73123 73103 3.70748 2.17188 0.0362895 0.0421283 2024-09-15T03:08:11.391 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 155 20 73416 73396 3.69833 2.28906 0.049969 0.0422328 2024-09-15T03:08:12.391 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 156 20 73704 73684 3.68905 2.25 0.0665006 0.0423398 2024-09-15T03:08:12.728 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:13.391 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 157 20 73988 73968 3.67968 2.21875 0.0355814 0.0424445 2024-09-15T03:08:14.056 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:14.075 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:14.114 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-15T03:08:12.693176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602151484,"num_objects":73509,"num_object_clones":0,"num_object_copies":294036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73536,"num_write_kb":588170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73563,"ondisk_log_size":73563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2277260,"kb_used_data":1048940,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375210100,"statfs":{"total":386547056640,"available":384215142400,"internally_reserved":0,"allocated":1074114560,"data_stored":1073481794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,5,10,8,12,12,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35274752,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":34448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003126"},"pg_stats":[{"pgid":"1.f","version":"31'4581","reported_seq":4623,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.235649+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:11.235649+0000","last_peered":"2024-09-15T03:08:11.235649+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:11.235649+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":36648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4434","reported_seq":4496,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.433767+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:11.433767+0000","last_peered":"2024-09-15T03:08:11.433767+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:11.433767+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36288114,"num_objects":4430,"num_object_clones":0,"num_object_copies":17720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":35452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4700","reported_seq":4779,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.132953+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:12.132953+0000","last_peered":"2024-09-15T03:08:12.132953+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:12.132953+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476601,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4478","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.446066+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:11.446066+0000","last_peered":"2024-09-15T03:08:11.446066+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:11.446066+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4478,"log_dups_size":0,"ondisk_log_size":4478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36648562,"num_objects":4474,"num_object_clones":0,"num_object_copies":17896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4476,"num_write_kb":35797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4632","reported_seq":4688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.331341+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:11.331341+0000","last_peered":"2024-09-15T03:08:11.331341+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:11.331341+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37918322,"num_objects":4629,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":37037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4645","reported_seq":4689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.236290+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:11.236290+0000","last_peered":"2024-09-15T03:08:11.236290+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:11.236290+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38051840,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":37160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4622","reported_seq":4686,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.487849+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:11.487849+0000","last_peered":"2024-09-15T03:08:11.487849+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:11.487849+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4622,"log_dups_size":0,"ondisk_log_size":4622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37811826,"num_objects":4616,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4620,"num_write_kb":36940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4658","reported_seq":4720,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.341252+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:11.341252+0000","last_peered":"2024-09-15T03:08:11.341252+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:11.341252+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38114930,"num_objects":4653,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":37236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4602","reported_seq":4659,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.133250+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:11.133250+0000","last_peered":"2024-09-15T03:08:11.133250+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:11.133250+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37629156,"num_objects":4594,"num_object_clones":0,"num_object_copies":18376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4599,"num_write_kb":36769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2552}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4608","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.270749+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:12.270749+0000","last_peered":"2024-09-15T03:08:12.270749+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:12.270749+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37729906,"num_objects":4606,"num_object_clones":0,"num_object_copies":18424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":36846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4618","reported_seq":4676,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.125540+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:11.125540+0000","last_peered":"2024-09-15T03:08:11.125540+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:11.125540+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37786027,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2535}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4586","reported_seq":6544,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.321261+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:11.321261+0000","last_peered":"2024-09-15T03:08:11.321261+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:11.321261+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37559097,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":36679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4649","reported_seq":4726,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.250685+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:12.250685+0000","last_peered":"2024-09-15T03:08:12.250685+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:12.250685+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4649,"log_dups_size":0,"ondisk_log_size":4649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38075193,"num_objects":4648,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":37183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2609}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4514","reported_seq":4564,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.169454+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:11.169454+0000","last_peered":"2024-09-15T03:08:11.169454+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:11.169454+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36943474,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4512,"num_write_kb":36092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4596","reported_seq":4677,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.231656+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:12.231656+0000","last_peered":"2024-09-15T03:08:12.231656+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:12.231656+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37623410,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4640","reported_seq":4703,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.408028+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:11.408028+0000","last_peered":"2024-09-15T03:08:11.408028+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:11.408028+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967474,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":37092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2553}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":602151484,"num_objects":73509,"num_object_clones":0,"num_object_copies":294036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73536,"num_write_kb":588170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1074946048,"data_stored":1074946048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73563,"ondisk_log_size":73563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591180,"kb_used_data":293316,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93780660,"statfs":{"total":96636764160,"available":96031395840,"internally_reserved":0,"allocated":300355584,"data_stored":300196606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,1,2,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":4.5830000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.38900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608392,"kb_used_data":294400,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93763448,"statfs":{"total":96636764160,"available":96013770752,"internally_reserved":0,"allocated":301465600,"data_stored":301309768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,1,0,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.1400000000000001},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.5569999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":1.591},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":1.417}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.372},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":2.0409999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473492,"kb_used_data":167308,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93898348,"statfs":{"total":96636764160,"available":96151908352,"internally_reserved":0,"allocated":171323392,"data_stored":171164414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,8,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.30399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.41899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604196,"kb_used_data":293916,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93767644,"statfs":{"total":96636764160,"available":96018067456,"internally_reserved":0,"allocated":300969984,"data_stored":300811006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.47799999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":1.323},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.21199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":6.0110000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":300883968,"data_stored":300883968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171675648,"data_stored":171675648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301752320,"data_stored":301752320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300634112,"data_stored":300634112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:14.115 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:14.391 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 158 20 74266 74246 3.67014 2.17188 0.0463571 0.0425599 2024-09-15T03:08:15.321 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:15.379 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:15.391 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 159 20 74540 74520 3.66052 2.14062 0.0476407 0.0426688 2024-09-15T03:08:15.395 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-15T03:08:14.693726+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":602151484,"num_objects":73509,"num_object_clones":0,"num_object_copies":294036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73536,"num_write_kb":588170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73563,"ondisk_log_size":73563,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2277260,"kb_used_data":1048940,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375210100,"statfs":{"total":386547056640,"available":384215142400,"internally_reserved":0,"allocated":1074114560,"data_stored":1073481794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,5,10,8,12,12,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23543808,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":22992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"1.f","version":"31'4581","reported_seq":4623,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.235649+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:11.235649+0000","last_peered":"2024-09-15T03:08:11.235649+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:11.235649+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4581,"log_dups_size":0,"ondisk_log_size":4581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37527552,"num_objects":4581,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4581,"num_write_kb":36648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4434","reported_seq":4496,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.433767+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:11.433767+0000","last_peered":"2024-09-15T03:08:11.433767+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:11.433767+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36288114,"num_objects":4430,"num_object_clones":0,"num_object_copies":17720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":35452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2432}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4700","reported_seq":4779,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.132953+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:12.132953+0000","last_peered":"2024-09-15T03:08:12.132953+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:12.132953+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476601,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4478","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.446066+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:11.446066+0000","last_peered":"2024-09-15T03:08:11.446066+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:11.446066+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4478,"log_dups_size":0,"ondisk_log_size":4478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36648562,"num_objects":4474,"num_object_clones":0,"num_object_copies":17896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4476,"num_write_kb":35797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2531}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4632","reported_seq":4688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.331341+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:11.331341+0000","last_peered":"2024-09-15T03:08:11.331341+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:11.331341+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37918322,"num_objects":4629,"num_object_clones":0,"num_object_copies":18516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":37037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4645","reported_seq":4689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.236290+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:11.236290+0000","last_peered":"2024-09-15T03:08:11.236290+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:11.236290+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38051840,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":37160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4622","reported_seq":4686,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.487849+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:11.487849+0000","last_peered":"2024-09-15T03:08:11.487849+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:11.487849+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4622,"log_dups_size":0,"ondisk_log_size":4622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37811826,"num_objects":4616,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4620,"num_write_kb":36940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4658","reported_seq":4720,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.341252+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:11.341252+0000","last_peered":"2024-09-15T03:08:11.341252+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:11.341252+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38114930,"num_objects":4653,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":37236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4602","reported_seq":4659,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.133250+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:11.133250+0000","last_peered":"2024-09-15T03:08:11.133250+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:11.133250+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37629156,"num_objects":4594,"num_object_clones":0,"num_object_copies":18376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4599,"num_write_kb":36769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2552}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4608","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.270749+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:12.270749+0000","last_peered":"2024-09-15T03:08:12.270749+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:12.270749+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37729906,"num_objects":4606,"num_object_clones":0,"num_object_copies":18424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":36846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4618","reported_seq":4676,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.125540+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:11.125540+0000","last_peered":"2024-09-15T03:08:11.125540+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:11.125540+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37786027,"num_objects":4613,"num_object_clones":0,"num_object_copies":18452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":36908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2535}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4586","reported_seq":6544,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.321261+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:11.321261+0000","last_peered":"2024-09-15T03:08:11.321261+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:11.321261+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37559097,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4585,"num_write_kb":36679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":961,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4649","reported_seq":4726,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.250685+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:12.250685+0000","last_peered":"2024-09-15T03:08:12.250685+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:12.250685+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4649,"log_dups_size":0,"ondisk_log_size":4649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38075193,"num_objects":4648,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":37183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2609}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4514","reported_seq":4564,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.169454+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:11.169454+0000","last_peered":"2024-09-15T03:08:11.169454+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:11.169454+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36943474,"num_objects":4510,"num_object_clones":0,"num_object_copies":18040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4512,"num_write_kb":36092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4596","reported_seq":4677,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.231656+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:12.231656+0000","last_peered":"2024-09-15T03:08:12.231656+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:12.231656+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37623410,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4640","reported_seq":4703,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:11.408028+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:11.408028+0000","last_peered":"2024-09-15T03:08:11.408028+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:11.408028+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37967474,"num_objects":4635,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":37092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2553}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":602151484,"num_objects":73509,"num_object_clones":0,"num_object_copies":294036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73536,"num_write_kb":588170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":7864320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1074946048,"data_stored":1074946048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73563,"ondisk_log_size":73563,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591180,"kb_used_data":293316,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93780660,"statfs":{"total":96636764160,"available":96031395840,"internally_reserved":0,"allocated":300355584,"data_stored":300196606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,1,2,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":4.5830000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.38900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608392,"kb_used_data":294400,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93763448,"statfs":{"total":96636764160,"available":96013770752,"internally_reserved":0,"allocated":301465600,"data_stored":301309768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,1,0,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.1400000000000001},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.5569999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":1.591},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":1.417}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.372},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":2.0409999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473492,"kb_used_data":167308,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93898348,"statfs":{"total":96636764160,"available":96151908352,"internally_reserved":0,"allocated":171323392,"data_stored":171164414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,8,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.30399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.41899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604196,"kb_used_data":293916,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93767644,"statfs":{"total":96636764160,"available":96018067456,"internally_reserved":0,"allocated":300969984,"data_stored":300811006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.47799999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":1.323},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.21199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":6.0110000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":300883968,"data_stored":300883968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171675648,"data_stored":171675648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301752320,"data_stored":301752320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300634112,"data_stored":300634112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:15.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4581", 'reported_seq': 4623, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.235649+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:08:11.235649+0000', 'last_peered': '2024-09-15T03:08:11.235649+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:08:11.235649+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4581, 'log_dups_size': 0, 'ondisk_log_size': 4581, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37527552, 'num_objects': 4581, 'num_object_clones': 0, 'num_object_copies': 18324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4581, 'num_write_kb': 36648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2605}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4434", 'reported_seq': 4496, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.433767+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:08:11.433767+0000', 'last_peered': '2024-09-15T03:08:11.433767+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:08:11.433767+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4434, 'log_dups_size': 0, 'ondisk_log_size': 4434, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36288114, 'num_objects': 4430, 'num_object_clones': 0, 'num_object_copies': 17720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4430, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4432, 'num_write_kb': 35452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2432}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4700", 'reported_seq': 4779, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:12.132953+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:08:12.132953+0000', 'last_peered': '2024-09-15T03:08:12.132953+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:08:12.132953+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4700, 'log_dups_size': 0, 'ondisk_log_size': 4700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38476601, 'num_objects': 4697, 'num_object_clones': 0, 'num_object_copies': 18788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4699, 'num_write_kb': 37582, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2595}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4478", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.446066+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:08:11.446066+0000', 'last_peered': '2024-09-15T03:08:11.446066+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:08:11.446066+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4478, 'log_dups_size': 0, 'ondisk_log_size': 4478, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36648562, 'num_objects': 4474, 'num_object_clones': 0, 'num_object_copies': 17896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4476, 'num_write_kb': 35797, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:08:15.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4632", 'reported_seq': 4688, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.331341+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:08:11.331341+0000', 'last_peered': '2024-09-15T03:08:11.331341+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:08:11.331341+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4632, 'log_dups_size': 0, 'ondisk_log_size': 4632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37918322, 'num_objects': 4629, 'num_object_clones': 0, 'num_object_copies': 18516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4630, 'num_write_kb': 37037, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2522}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4645", 'reported_seq': 4689, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.236290+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:08:11.236290+0000', 'last_peered': '2024-09-15T03:08:11.236290+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:08:11.236290+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4645, 'log_dups_size': 0, 'ondisk_log_size': 4645, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38051840, 'num_objects': 4645, 'num_object_clones': 0, 'num_object_copies': 18580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4645, 'num_write_kb': 37160, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2589}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4622", 'reported_seq': 4686, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.487849+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:08:11.487849+0000', 'last_peered': '2024-09-15T03:08:11.487849+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:08:11.487849+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4622, 'log_dups_size': 0, 'ondisk_log_size': 4622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37811826, 'num_objects': 4616, 'num_object_clones': 0, 'num_object_copies': 18464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4620, 'num_write_kb': 36940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2564}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4658", 'reported_seq': 4720, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.341252+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:08:11.341252+0000', 'last_peered': '2024-09-15T03:08:11.341252+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:08:11.341252+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4658, 'log_dups_size': 0, 'ondisk_log_size': 4658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38114930, 'num_objects': 4653, 'num_object_clones': 0, 'num_object_copies': 18612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4656, 'num_write_kb': 37236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4602", 'reported_seq': 4659, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.133250+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:08:11.133250+0000', 'last_peered': '2024-09-15T03:08:11.133250+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:08:11.133250+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4602, 'log_dups_size': 0, 'ondisk_log_size': 4602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37629156, 'num_objects': 4594, 'num_object_clones': 0, 'num_object_copies': 18376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4599, 'num_write_kb': 36769, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2552}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4608", 'reported_seq': 4691, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:12.270749+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:08:12.270749+0000', 'last_peered': '2024-09-15T03:08:12.270749+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:08:12.270749+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4608, 'log_dups_size': 0, 'ondisk_log_size': 4608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37729906, 'num_objects': 4606, 'num_object_clones': 0, 'num_object_copies': 18424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4606, 'num_write_kb': 36846, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2548}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:08:15.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4618", 'reported_seq': 4676, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.125540+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:08:11.125540+0000', 'last_peered': '2024-09-15T03:08:11.125540+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:08:11.125540+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4618, 'log_dups_size': 0, 'ondisk_log_size': 4618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37786027, 'num_objects': 4613, 'num_object_clones': 0, 'num_object_copies': 18452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4615, 'num_write_kb': 36908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2535}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4586", 'reported_seq': 6544, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.321261+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:08:11.321261+0000', 'last_peered': '2024-09-15T03:08:11.321261+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:08:11.321261+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4586, 'log_dups_size': 0, 'ondisk_log_size': 4586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37559097, 'num_objects': 4585, 'num_object_clones': 0, 'num_object_copies': 18340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1032, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4585, 'num_write_kb': 36679, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 961, 'num_bytes_recovered': 7864320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4649", 'reported_seq': 4726, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:12.250685+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:08:12.250685+0000', 'last_peered': '2024-09-15T03:08:12.250685+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:08:12.250685+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4649, 'log_dups_size': 0, 'ondisk_log_size': 4649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38075193, 'num_objects': 4648, 'num_object_clones': 0, 'num_object_copies': 18592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4648, 'num_write_kb': 37183, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2609}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4514", 'reported_seq': 4564, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.169454+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:08:11.169454+0000', 'last_peered': '2024-09-15T03:08:11.169454+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:08:11.169454+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4514, 'log_dups_size': 0, 'ondisk_log_size': 4514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36943474, 'num_objects': 4510, 'num_object_clones': 0, 'num_object_copies': 18040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4512, 'num_write_kb': 36092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2474}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4596", 'reported_seq': 4677, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:12.231656+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:08:12.231656+0000', 'last_peered': '2024-09-15T03:08:12.231656+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:08:12.231656+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4596, 'log_dups_size': 0, 'ondisk_log_size': 4596, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37623410, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 18372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4594, 'num_write_kb': 36749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2605}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:08:15.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4640", 'reported_seq': 4703, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:11.408028+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:08:11.408028+0000', 'last_peered': '2024-09-15T03:08:11.408028+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:08:11.408028+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4640, 'log_dups_size': 0, 'ondisk_log_size': 4640, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37967474, 'num_objects': 4635, 'num_object_clones': 0, 'num_object_copies': 18540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4635, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4638, 'num_write_kb': 37092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:15.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:08:16.391 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:08:16.391228+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0427882 2024-09-15T03:08:16.392 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:08:16.392 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 160 20 74799 74779 3.65028 2.02344 0.109121 0.0427882 2024-09-15T03:08:16.914 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:16.914 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31696/294036 objects degraded (10.780%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":219,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73509,"data_bytes":602151484,"bytes_used":2331914240,"bytes_avail":384215142400,"bytes_total":386547056640,"degraded_objects":31696,"degraded_total":294036,"degraded_ratio":0.10779632425961447,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":51868,"recovering_keys_per_sec":0,"num_objects_recovered":76,"num_bytes_recovered":622592,"num_keys_recovered":0,"write_bytes_sec":1961461,"read_op_per_sec":0,"write_op_per_sec":239},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:08:16.943 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:08:16.943 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:17.392 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 161 20 75067 75047 3.64062 2.09375 0.100973 0.0429021 2024-09-15T03:08:18.392 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 162 20 75330 75310 3.63083 2.05469 0.0528519 0.0430157 2024-09-15T03:08:18.631 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:18.677 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:18.688 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-15T03:08:16.696472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":610638396,"num_objects":74545,"num_object_clones":0,"num_object_copies":298180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74572,"num_write_kb":596458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1041,"num_bytes_recovered":8306688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74599,"ondisk_log_size":74599,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2284260,"kb_used_data":1060100,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375203100,"statfs":{"total":386547056640,"available":384207974400,"internally_reserved":0,"allocated":1085542400,"data_stored":1084909634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,7,9,16,12,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32030720,"num_objects":3910,"num_object_clones":0,"num_object_copies":15640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3910,"num_write_kb":31280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005362"},"pg_stats":[{"pgid":"1.f","version":"31'4670","reported_seq":4712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.265803+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:16.265803+0000","last_peered":"2024-09-15T03:08:16.265803+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:16.265803+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4670,"log_dups_size":0,"ondisk_log_size":4670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38256640,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4670,"num_write_kb":37360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2694}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4517","reported_seq":4579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.327870+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:16.327870+0000","last_peered":"2024-09-15T03:08:16.327870+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:16.327870+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4517,"log_dups_size":0,"ondisk_log_size":4517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36968050,"num_objects":4513,"num_object_clones":0,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4515,"num_write_kb":36116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2515}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4700","reported_seq":4779,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.132953+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:12.132953+0000","last_peered":"2024-09-15T03:08:12.132953+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:12.132953+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38476601,"num_objects":4697,"num_object_clones":0,"num_object_copies":18788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":37582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4565","reported_seq":4628,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.506044+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:16.506044+0000","last_peered":"2024-09-15T03:08:16.506044+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:16.506044+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4565,"log_dups_size":0,"ondisk_log_size":4565,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37361266,"num_objects":4561,"num_object_clones":0,"num_object_copies":18244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4563,"num_write_kb":36493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2618}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4729","reported_seq":4785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.303005+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:16.303005+0000","last_peered":"2024-09-15T03:08:16.303005+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:16.303005+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38712946,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4727,"num_write_kb":37813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2619}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4732","reported_seq":4775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.308237+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:16.308237+0000","last_peered":"2024-09-15T03:08:16.308237+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:16.308237+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38764544,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4732,"num_write_kb":37856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2676}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4708","reported_seq":4772,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.496481+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:16.496481+0000","last_peered":"2024-09-15T03:08:16.496481+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:16.496481+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38516338,"num_objects":4702,"num_object_clones":0,"num_object_copies":18808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4706,"num_write_kb":37628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2650}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4744","reported_seq":4806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.376548+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:16.376548+0000","last_peered":"2024-09-15T03:08:16.376548+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:16.376548+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4744,"log_dups_size":0,"ondisk_log_size":4744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38819442,"num_objects":4739,"num_object_clones":0,"num_object_copies":18956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":37924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4684","reported_seq":4741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.302951+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:16.302951+0000","last_peered":"2024-09-15T03:08:16.302951+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:16.302951+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4684,"log_dups_size":0,"ondisk_log_size":4684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38300900,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4681,"num_write_kb":37425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2634}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4608","reported_seq":4691,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.270749+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:12.270749+0000","last_peered":"2024-09-15T03:08:12.270749+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:12.270749+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37729906,"num_objects":4606,"num_object_clones":0,"num_object_copies":18424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":36846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2548}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4710","reported_seq":4768,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.384511+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:16.384511+0000","last_peered":"2024-09-15T03:08:16.384511+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:16.384511+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4710,"log_dups_size":0,"ondisk_log_size":4710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38539691,"num_objects":4705,"num_object_clones":0,"num_object_copies":18820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4662","reported_seq":6728,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.358768+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:16.358768+0000","last_peered":"2024-09-15T03:08:16.358768+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:16.358768+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38181689,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":37287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1015,"num_bytes_recovered":8306688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3683}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4649","reported_seq":4726,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.250685+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:12.250685+0000","last_peered":"2024-09-15T03:08:12.250685+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:12.250685+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4649,"log_dups_size":0,"ondisk_log_size":4649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38075193,"num_objects":4648,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":37183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2609}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4590","reported_seq":4640,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.309790+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:16.309790+0000","last_peered":"2024-09-15T03:08:16.309790+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:16.309790+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37566066,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4588,"num_write_kb":36700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4596","reported_seq":4677,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:12.231656+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:12.231656+0000","last_peered":"2024-09-15T03:08:12.231656+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:12.231656+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37623410,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":36749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2605}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4735","reported_seq":4796,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:16.535665+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:16.535665+0000","last_peered":"2024-09-15T03:08:16.535665+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:16.535665+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745714,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":610638396,"num_objects":74545,"num_object_clones":0,"num_object_copies":298180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74572,"num_write_kb":596458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1041,"num_bytes_recovered":8306688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1086291968,"data_stored":1086291968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74599,"ondisk_log_size":74599,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605116,"kb_used_data":298932,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93766724,"statfs":{"total":96636764160,"available":96017125376,"internally_reserved":0,"allocated":306106368,"data_stored":305947390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,2,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":16.783999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.49099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":1.427}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608392,"kb_used_data":294400,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93763448,"statfs":{"total":96636764160,"available":96013770752,"internally_reserved":0,"allocated":301465600,"data_stored":301309768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,1,0,4,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:20 2024","interfaces":[{"interface":"back","average":{"1min":3.3519999999999999,"5min":1.248,"15min":0.89700000000000002},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":23.280000000000001,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.1400000000000001},{"interface":"front","average":{"1min":1.671,"5min":0.99099999999999999,"15min":0.877},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":19.172999999999998,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.5569999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:07:08 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":1.591},{"interface":"front","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.5919999999999996,"5min":6.5919999999999996,"15min":6.5919999999999996},"last":1.417}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.372},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":2.0409999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473492,"kb_used_data":167308,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93898348,"statfs":{"total":96636764160,"available":96151908352,"internally_reserved":0,"allocated":171323392,"data_stored":171164414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,8,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.30399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.41899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597260,"kb_used_data":299460,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93774580,"statfs":{"total":96636764160,"available":96025169920,"internally_reserved":0,"allocated":306647040,"data_stored":306488062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1,0,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":4.5709999999999997},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":27.736999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":30.802},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.39500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306536448,"data_stored":306536448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171675648,"data_stored":171675648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301752320,"data_stored":301752320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":306327552,"data_stored":306327552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:19.392 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 163 20 75612 75592 3.62207 2.20312 0.119847 0.0431232 2024-09-15T03:08:20.392 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 164 19 75889 75870 3.61322 2.17188 0.0354061 0.043223 2024-09-15T03:08:21.393 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 165 20 76173 76153 3.6047 2.21094 0.179081 0.0433273 2024-09-15T03:08:21.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:22.393 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 166 20 76446 76426 3.59583 2.13281 0.0471884 0.0434298 2024-09-15T03:08:22.826 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:22.849 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:22.861 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-15T03:08:22.700892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":624769596,"num_objects":76270,"num_object_clones":0,"num_object_copies":305080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76297,"num_write_kb":610258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":8675328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76324,"ondisk_log_size":76324,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2317704,"kb_used_data":1093544,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375169656,"statfs":{"total":386547056640,"available":384173727744,"internally_reserved":0,"allocated":1119789056,"data_stored":1119156290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,9,7,12,11,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34136064,"num_objects":4167,"num_object_clones":0,"num_object_copies":16668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4167,"num_write_kb":33336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008187"},"pg_stats":[{"pgid":"1.f","version":"31'4766","reported_seq":4808,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.284003+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:21.284003+0000","last_peered":"2024-09-15T03:08:21.284003+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:21.284003+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39043072,"num_objects":4766,"num_object_clones":0,"num_object_copies":19064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4766,"num_write_kb":38128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4595","reported_seq":4657,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.450306+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:21.450306+0000","last_peered":"2024-09-15T03:08:21.450306+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:21.450306+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37607026,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":36740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4887","reported_seq":4968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.280654+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:22.280654+0000","last_peered":"2024-09-15T03:08:22.280654+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:22.280654+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40008505,"num_objects":4884,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":39078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4637","reported_seq":4699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.463424+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:21.463424+0000","last_peered":"2024-09-15T03:08:21.463424+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:21.463424+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37951090,"num_objects":4633,"num_object_clones":0,"num_object_copies":18532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":37069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4811","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.410768+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:21.410768+0000","last_peered":"2024-09-15T03:08:21.410768+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:21.410768+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4811,"log_dups_size":0,"ondisk_log_size":4811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39384690,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4809,"num_write_kb":38469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4813","reported_seq":4857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.408607+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:21.408607+0000","last_peered":"2024-09-15T03:08:21.408607+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:21.408607+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39428096,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4795","reported_seq":4860,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.513863+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:21.513863+0000","last_peered":"2024-09-15T03:08:21.513863+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:21.513863+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39229042,"num_objects":4789,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4793,"num_write_kb":38324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2737}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4839","reported_seq":4902,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.476607+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:21.476607+0000","last_peered":"2024-09-15T03:08:21.476607+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:21.476607+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39597682,"num_objects":4834,"num_object_clones":0,"num_object_copies":19336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4837,"num_write_kb":38684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2753}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4773","reported_seq":4830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.398838+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:21.398838+0000","last_peered":"2024-09-15T03:08:21.398838+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:21.398838+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39029988,"num_objects":4765,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4770,"num_write_kb":38137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4782","reported_seq":4865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.099726+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:22.099726+0000","last_peered":"2024-09-15T03:08:22.099726+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:22.099726+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39155314,"num_objects":4780,"num_object_clones":0,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":38238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2722}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4792","reported_seq":4850,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.405094+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:21.405094+0000","last_peered":"2024-09-15T03:08:21.405094+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:21.405094+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39211435,"num_objects":4787,"num_object_clones":0,"num_object_copies":19148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4743","reported_seq":6899,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.409654+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:21.409654+0000","last_peered":"2024-09-15T03:08:21.409654+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:21.409654+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38845241,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":37935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1060,"num_bytes_recovered":8675328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4842","reported_seq":4918,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.200246+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:22.200246+0000","last_peered":"2024-09-15T03:08:22.200246+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:22.200246+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39656249,"num_objects":4841,"num_object_clones":0,"num_object_copies":19364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4841,"num_write_kb":38727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4685","reported_seq":4735,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.336504+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:21.336504+0000","last_peered":"2024-09-15T03:08:21.336504+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:21.336504+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38344306,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4751","reported_seq":4832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.181849+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:22.181849+0000","last_peered":"2024-09-15T03:08:22.181849+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:22.181849+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38893170,"num_objects":4748,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":37989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4813","reported_seq":4876,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.497822+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:21.497822+0000","last_peered":"2024-09-15T03:08:21.497822+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:21.497822+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39384690,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4811,"num_write_kb":38476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":624769596,"num_objects":76270,"num_object_clones":0,"num_object_copies":305080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76297,"num_write_kb":610258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":8675328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1120919552,"data_stored":1120919552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76324,"ondisk_log_size":76324,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614880,"kb_used_data":304536,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93756960,"statfs":{"total":96636764160,"available":96007127040,"internally_reserved":0,"allocated":311844864,"data_stored":311685886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,3,2,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.411}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":3.0089999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.54400000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":16.216999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.52200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610908,"kb_used_data":305236,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93760932,"statfs":{"total":96636764160,"available":96011194368,"internally_reserved":0,"allocated":312561664,"data_stored":312405832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":5.0979999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":7.0899999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":3.5819999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":2.6269999999999998},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":2.581}]}]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480764,"kb_used_data":178740,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93891076,"statfs":{"total":96636764160,"available":96144461824,"internally_reserved":0,"allocated":183029760,"data_stored":182870782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.27600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.31},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.60299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.48599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611152,"kb_used_data":305032,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93760688,"statfs":{"total":96636764160,"available":96010944512,"internally_reserved":0,"allocated":312352768,"data_stored":312193790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,2,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":3.0510000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":3.121},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.35499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":4.5869999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312201216,"data_stored":312201216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183570432,"data_stored":183570432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313081856,"data_stored":313081856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312066048,"data_stored":312066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:22.862 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:23.393 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 167 20 76722 76702 3.58721 2.15625 0.0635424 0.0435376 2024-09-15T03:08:24.290 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:24.292 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:24.308 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-15T03:08:22.700892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":624769596,"num_objects":76270,"num_object_clones":0,"num_object_copies":305080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76297,"num_write_kb":610258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":8675328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76324,"ondisk_log_size":76324,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2317704,"kb_used_data":1093544,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375169656,"statfs":{"total":386547056640,"available":384173727744,"internally_reserved":0,"allocated":1119789056,"data_stored":1119156290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,9,7,12,11,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34136064,"num_objects":4167,"num_object_clones":0,"num_object_copies":16668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4167,"num_write_kb":33336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008187"},"pg_stats":[{"pgid":"1.f","version":"31'4766","reported_seq":4808,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.284003+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:21.284003+0000","last_peered":"2024-09-15T03:08:21.284003+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:21.284003+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39043072,"num_objects":4766,"num_object_clones":0,"num_object_copies":19064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4766,"num_write_kb":38128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2790}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4595","reported_seq":4657,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.450306+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:21.450306+0000","last_peered":"2024-09-15T03:08:21.450306+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:21.450306+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4595,"log_dups_size":0,"ondisk_log_size":4595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37607026,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":36740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4887","reported_seq":4968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.280654+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:22.280654+0000","last_peered":"2024-09-15T03:08:22.280654+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:22.280654+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40008505,"num_objects":4884,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":39078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4637","reported_seq":4699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.463424+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:21.463424+0000","last_peered":"2024-09-15T03:08:21.463424+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:21.463424+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37951090,"num_objects":4633,"num_object_clones":0,"num_object_copies":18532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":37069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2690}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4811","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.410768+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:21.410768+0000","last_peered":"2024-09-15T03:08:21.410768+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:21.410768+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4811,"log_dups_size":0,"ondisk_log_size":4811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39384690,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4809,"num_write_kb":38469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4813","reported_seq":4857,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.408607+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:21.408607+0000","last_peered":"2024-09-15T03:08:21.408607+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:21.408607+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39428096,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2757}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4795","reported_seq":4860,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.513863+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:21.513863+0000","last_peered":"2024-09-15T03:08:21.513863+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:21.513863+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39229042,"num_objects":4789,"num_object_clones":0,"num_object_copies":19156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4793,"num_write_kb":38324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2737}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4839","reported_seq":4902,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.476607+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:21.476607+0000","last_peered":"2024-09-15T03:08:21.476607+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:21.476607+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39597682,"num_objects":4834,"num_object_clones":0,"num_object_copies":19336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4837,"num_write_kb":38684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2753}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4773","reported_seq":4830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.398838+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:21.398838+0000","last_peered":"2024-09-15T03:08:21.398838+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:21.398838+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39029988,"num_objects":4765,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4770,"num_write_kb":38137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4782","reported_seq":4865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.099726+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:22.099726+0000","last_peered":"2024-09-15T03:08:22.099726+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:22.099726+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39155314,"num_objects":4780,"num_object_clones":0,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":38238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2722}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4792","reported_seq":4850,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.405094+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:21.405094+0000","last_peered":"2024-09-15T03:08:21.405094+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:21.405094+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39211435,"num_objects":4787,"num_object_clones":0,"num_object_copies":19148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":38300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4743","reported_seq":6899,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.409654+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:21.409654+0000","last_peered":"2024-09-15T03:08:21.409654+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:21.409654+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38845241,"num_objects":4742,"num_object_clones":0,"num_object_copies":18968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4742,"num_write_kb":37935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1060,"num_bytes_recovered":8675328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4842","reported_seq":4918,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.200246+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:22.200246+0000","last_peered":"2024-09-15T03:08:22.200246+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:22.200246+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39656249,"num_objects":4841,"num_object_clones":0,"num_object_copies":19364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4841,"num_write_kb":38727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4685","reported_seq":4735,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.336504+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:21.336504+0000","last_peered":"2024-09-15T03:08:21.336504+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:21.336504+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38344306,"num_objects":4681,"num_object_clones":0,"num_object_copies":18724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":37460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4751","reported_seq":4832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.181849+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:22.181849+0000","last_peered":"2024-09-15T03:08:22.181849+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:22.181849+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38893170,"num_objects":4748,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":37989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4813","reported_seq":4876,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:21.497822+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:21.497822+0000","last_peered":"2024-09-15T03:08:21.497822+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:21.497822+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39384690,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4811,"num_write_kb":38476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":624769596,"num_objects":76270,"num_object_clones":0,"num_object_copies":305080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76297,"num_write_kb":610258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1086,"num_bytes_recovered":8675328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1120919552,"data_stored":1120919552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76324,"ondisk_log_size":76324,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614880,"kb_used_data":304536,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93756960,"statfs":{"total":96636764160,"available":96007127040,"internally_reserved":0,"allocated":311844864,"data_stored":311685886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,3,2,5,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.411}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":3.0089999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.54400000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":16.216999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.52200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610908,"kb_used_data":305236,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93760932,"statfs":{"total":96636764160,"available":96011194368,"internally_reserved":0,"allocated":312561664,"data_stored":312405832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":5.0979999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":7.0899999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":3.5819999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":2.6269999999999998},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":2.581}]}]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480764,"kb_used_data":178740,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93891076,"statfs":{"total":96636764160,"available":96144461824,"internally_reserved":0,"allocated":183029760,"data_stored":182870782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.27600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.31},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.60299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.48599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611152,"kb_used_data":305032,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93760688,"statfs":{"total":96636764160,"available":96010944512,"internally_reserved":0,"allocated":312352768,"data_stored":312193790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,2,5,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":3.0510000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":3.121},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.35499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.39700000000000002},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":4.5869999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312201216,"data_stored":312201216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183570432,"data_stored":183570432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313081856,"data_stored":313081856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312066048,"data_stored":312066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:24.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:08:24.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4766", 'reported_seq': 4808, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.284003+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:08:21.284003+0000', 'last_peered': '2024-09-15T03:08:21.284003+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:08:21.284003+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4766, 'log_dups_size': 0, 'ondisk_log_size': 4766, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39043072, 'num_objects': 4766, 'num_object_clones': 0, 'num_object_copies': 19064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4766, 'num_write_kb': 38128, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2790}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:24.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:08:24.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4595", 'reported_seq': 4657, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.450306+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:08:21.450306+0000', 'last_peered': '2024-09-15T03:08:21.450306+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:08:21.450306+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4595, 'log_dups_size': 0, 'ondisk_log_size': 4595, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37607026, 'num_objects': 4591, 'num_object_clones': 0, 'num_object_copies': 18364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 36740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4887", 'reported_seq': 4968, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:22.280654+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:08:22.280654+0000', 'last_peered': '2024-09-15T03:08:22.280654+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:08:22.280654+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4887, 'log_dups_size': 0, 'ondisk_log_size': 4887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40008505, 'num_objects': 4884, 'num_object_clones': 0, 'num_object_copies': 19536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4886, 'num_write_kb': 39078, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2782}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4637", 'reported_seq': 4699, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.463424+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:08:21.463424+0000', 'last_peered': '2024-09-15T03:08:21.463424+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:08:21.463424+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37951090, 'num_objects': 4633, 'num_object_clones': 0, 'num_object_copies': 18532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4635, 'num_write_kb': 37069, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4811", 'reported_seq': 4867, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.410768+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:08:21.410768+0000', 'last_peered': '2024-09-15T03:08:21.410768+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:08:21.410768+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4811, 'log_dups_size': 0, 'ondisk_log_size': 4811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39384690, 'num_objects': 4808, 'num_object_clones': 0, 'num_object_copies': 19232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4809, 'num_write_kb': 38469, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2701}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4813", 'reported_seq': 4857, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.408607+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:08:21.408607+0000', 'last_peered': '2024-09-15T03:08:21.408607+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:08:21.408607+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4813, 'log_dups_size': 0, 'ondisk_log_size': 4813, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39428096, 'num_objects': 4813, 'num_object_clones': 0, 'num_object_copies': 19252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4813, 'num_write_kb': 38504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2757}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:08:24.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4795", 'reported_seq': 4860, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.513863+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:08:21.513863+0000', 'last_peered': '2024-09-15T03:08:21.513863+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:08:21.513863+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4795, 'log_dups_size': 0, 'ondisk_log_size': 4795, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39229042, 'num_objects': 4789, 'num_object_clones': 0, 'num_object_copies': 19156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4793, 'num_write_kb': 38324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2737}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4839", 'reported_seq': 4902, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.476607+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:08:21.476607+0000', 'last_peered': '2024-09-15T03:08:21.476607+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:08:21.476607+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4839, 'log_dups_size': 0, 'ondisk_log_size': 4839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39597682, 'num_objects': 4834, 'num_object_clones': 0, 'num_object_copies': 19336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4837, 'num_write_kb': 38684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2753}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4773", 'reported_seq': 4830, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.398838+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:08:21.398838+0000', 'last_peered': '2024-09-15T03:08:21.398838+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:08:21.398838+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4773, 'log_dups_size': 0, 'ondisk_log_size': 4773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39029988, 'num_objects': 4765, 'num_object_clones': 0, 'num_object_copies': 19060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4770, 'num_write_kb': 38137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2723}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4782", 'reported_seq': 4865, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:22.099726+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:08:22.099726+0000', 'last_peered': '2024-09-15T03:08:22.099726+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:08:22.099726+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4782, 'log_dups_size': 0, 'ondisk_log_size': 4782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39155314, 'num_objects': 4780, 'num_object_clones': 0, 'num_object_copies': 19120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4780, 'num_write_kb': 38238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2722}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4792", 'reported_seq': 4850, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.405094+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:08:21.405094+0000', 'last_peered': '2024-09-15T03:08:21.405094+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:08:21.405094+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4792, 'log_dups_size': 0, 'ondisk_log_size': 4792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39211435, 'num_objects': 4787, 'num_object_clones': 0, 'num_object_copies': 19148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4789, 'num_write_kb': 38300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2709}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:08:24.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4743", 'reported_seq': 6899, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.409654+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:08:21.409654+0000', 'last_peered': '2024-09-15T03:08:21.409654+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:08:21.409654+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4743, 'log_dups_size': 0, 'ondisk_log_size': 4743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38845241, 'num_objects': 4742, 'num_object_clones': 0, 'num_object_copies': 18968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 933, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4742, 'num_write_kb': 37935, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1060, 'num_bytes_recovered': 8675328, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3809}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:24.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:08:24.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4842", 'reported_seq': 4918, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:22.200246+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:08:22.200246+0000', 'last_peered': '2024-09-15T03:08:22.200246+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:08:22.200246+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4842, 'log_dups_size': 0, 'ondisk_log_size': 4842, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39656249, 'num_objects': 4841, 'num_object_clones': 0, 'num_object_copies': 19364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4841, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4841, 'num_write_kb': 38727, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2802}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:24.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:08:24.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4685", 'reported_seq': 4735, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.336504+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:08:21.336504+0000', 'last_peered': '2024-09-15T03:08:21.336504+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:08:21.336504+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4685, 'log_dups_size': 0, 'ondisk_log_size': 4685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38344306, 'num_objects': 4681, 'num_object_clones': 0, 'num_object_copies': 18724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4683, 'num_write_kb': 37460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2645}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:24.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:08:24.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4751", 'reported_seq': 4832, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:22.181849+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:08:22.181849+0000', 'last_peered': '2024-09-15T03:08:22.181849+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:08:22.181849+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4751, 'log_dups_size': 0, 'ondisk_log_size': 4751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38893170, 'num_objects': 4748, 'num_object_clones': 0, 'num_object_copies': 18992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4749, 'num_write_kb': 37989, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:24.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:08:24.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4813", 'reported_seq': 4876, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:21.497822+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:08:21.497822+0000', 'last_peered': '2024-09-15T03:08:21.497822+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:08:21.497822+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4813, 'log_dups_size': 0, 'ondisk_log_size': 4813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39384690, 'num_objects': 4808, 'num_object_clones': 0, 'num_object_copies': 19232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4811, 'num_write_kb': 38476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:24.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:08:24.395 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 168 20 76986 76966 3.57811 2.0625 0.0271216 0.0436525 2024-09-15T03:08:25.400 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 169 20 77251 77231 3.56915 2.07031 0.0787772 0.043762 2024-09-15T03:08:25.752 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:25.752 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31597/305080 objects degraded (10.357%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":228,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76270,"data_bytes":624769596,"bytes_used":2373328896,"bytes_avail":384173727744,"bytes_total":386547056640,"degraded_objects":31597,"degraded_total":305080,"degraded_ratio":0.10356955552641929,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":67537,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"write_bytes_sec":1883546,"read_op_per_sec":0,"write_op_per_sec":229},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:08:25.814 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:08:25.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:26.397 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 170 20 77516 77496 3.56034 2.07031 0.0695496 0.0438631 2024-09-15T03:08:27.248 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:27.269 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:27.333 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-15T03:08:26.704335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633117244,"num_objects":77289,"num_object_clones":0,"num_object_copies":309156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77316,"num_write_kb":618410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77343,"ondisk_log_size":77343,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2324408,"kb_used_data":1104408,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375162952,"statfs":{"total":386547056640,"available":384166862848,"internally_reserved":0,"allocated":1130913792,"data_stored":1130281026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,9,9,11,16,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30965760,"num_objects":3780,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":30240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010609"},"pg_stats":[{"pgid":"1.f","version":"31'4844","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.318801+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:26.318801+0000","last_peered":"2024-09-15T03:08:26.318801+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:26.318801+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39682048,"num_objects":4844,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4667","reported_seq":4729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.456557+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:26.456557+0000","last_peered":"2024-09-15T03:08:26.456557+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:26.456557+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196850,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4665,"num_write_kb":37316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4887","reported_seq":4968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.280654+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:22.280654+0000","last_peered":"2024-09-15T03:08:22.280654+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:22.280654+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40008505,"num_objects":4884,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":39078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4740","reported_seq":4803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.522703+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:26.522703+0000","last_peered":"2024-09-15T03:08:26.522703+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:26.522703+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38794866,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4738,"num_write_kb":37893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4912","reported_seq":4968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.355020+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:26.355020+0000","last_peered":"2024-09-15T03:08:26.355020+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:26.355020+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40212082,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4893","reported_seq":4937,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.232848+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:26.232848+0000","last_peered":"2024-09-15T03:08:26.232848+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:26.232848+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4893,"log_dups_size":0,"ondisk_log_size":4893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40083456,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4893,"num_write_kb":39144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4880","reported_seq":4944,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.576776+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:26.576776+0000","last_peered":"2024-09-15T03:08:26.576776+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:26.576776+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39925362,"num_objects":4874,"num_object_clones":0,"num_object_copies":19496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4878,"num_write_kb":39004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2822}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4941","reported_seq":5004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.468660+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:26.468660+0000","last_peered":"2024-09-15T03:08:26.468660+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:26.468660+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40433266,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":39500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4855","reported_seq":4912,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.398686+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:26.398686+0000","last_peered":"2024-09-15T03:08:26.398686+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:26.398686+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39701732,"num_objects":4847,"num_object_clones":0,"num_object_copies":19388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4782","reported_seq":4865,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.099726+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:22.099726+0000","last_peered":"2024-09-15T03:08:22.099726+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:22.099726+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4782,"log_dups_size":0,"ondisk_log_size":4782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39155314,"num_objects":4780,"num_object_clones":0,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4780,"num_write_kb":38238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2722}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4868","reported_seq":4926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.378787+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:26.378787+0000","last_peered":"2024-09-15T03:08:26.378787+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:26.378787+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4868,"log_dups_size":0,"ondisk_log_size":4868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39834027,"num_objects":4863,"num_object_clones":0,"num_object_copies":19452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":38908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4830","reported_seq":7086,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.468002+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:26.468002+0000","last_peered":"2024-09-15T03:08:26.468002+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:26.468002+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39557945,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3946}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4842","reported_seq":4918,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.200246+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:22.200246+0000","last_peered":"2024-09-15T03:08:22.200246+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:22.200246+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39656249,"num_objects":4841,"num_object_clones":0,"num_object_copies":19364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4841,"num_write_kb":38727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2802}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4760","reported_seq":4810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.407816+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:26.407816+0000","last_peered":"2024-09-15T03:08:26.407816+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:26.407816+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958706,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4758,"num_write_kb":38060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2720}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4751","reported_seq":4832,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:22.181849+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:22.181849+0000","last_peered":"2024-09-15T03:08:22.181849+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:22.181849+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38893170,"num_objects":4748,"num_object_clones":0,"num_object_copies":18992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":37989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4891","reported_seq":4954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.462048+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:26.462048+0000","last_peered":"2024-09-15T03:08:26.462048+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:26.462048+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40023666,"num_objects":4886,"num_object_clones":0,"num_object_copies":19544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":39100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2804}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":633117244,"num_objects":77289,"num_object_clones":0,"num_object_copies":309156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77316,"num_write_kb":618410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1131950080,"data_stored":1131950080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77343,"ondisk_log_size":77343,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611952,"kb_used_data":309928,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93759888,"statfs":{"total":96636764160,"available":96010125312,"internally_reserved":0,"allocated":317366272,"data_stored":317207294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,6,8,11,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.32900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.314}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.28499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610908,"kb_used_data":305236,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93760932,"statfs":{"total":96636764160,"available":96011194368,"internally_reserved":0,"allocated":312561664,"data_stored":312405832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.26800000000000002},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":5.0979999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":7.0899999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":3.5819999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":2.6269999999999998},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":2.581}]}]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480764,"kb_used_data":178740,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93891076,"statfs":{"total":96636764160,"available":96144461824,"internally_reserved":0,"allocated":183029760,"data_stored":182870782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.27600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.31},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.60299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.48599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620784,"kb_used_data":310504,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93751056,"statfs":{"total":96636764160,"available":96001081344,"internally_reserved":0,"allocated":317956096,"data_stored":317797118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.247},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":14.301}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":6.3630000000000004},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317706240,"data_stored":317706240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183570432,"data_stored":183570432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313081856,"data_stored":313081856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":317591552,"data_stored":317591552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:27.397 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 171 20 77785 77765 3.55181 2.10156 0.0429919 0.0439789 2024-09-15T03:08:28.397 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 172 20 78065 78045 3.54387 2.1875 0.0883254 0.0440762 2024-09-15T03:08:29.397 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 173 20 78340 78320 3.5358 2.14844 0.0653777 0.0441743 2024-09-15T03:08:30.335 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:30.397 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 174 20 78617 78597 3.52792 2.16406 0.0219691 0.0442757 2024-09-15T03:08:31.398 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 175 20 78878 78858 3.51941 2.03906 0.14882 0.044374 2024-09-15T03:08:31.557 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:31.585 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:31.607 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-15T03:08:30.711511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635853372,"num_objects":77623,"num_object_clones":0,"num_object_copies":310492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77650,"num_write_kb":621082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77677,"ondisk_log_size":77677,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2343728,"kb_used_data":1115408,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375143632,"statfs":{"total":386547056640,"available":384147079168,"internally_reserved":0,"allocated":1142177792,"data_stored":1141545026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,5,8,12,21,11,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22364160,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012220"},"pg_stats":[{"pgid":"1.f","version":"31'4844","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.318801+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:26.318801+0000","last_peered":"2024-09-15T03:08:26.318801+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:26.318801+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39682048,"num_objects":4844,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4667","reported_seq":4729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.456557+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:26.456557+0000","last_peered":"2024-09-15T03:08:26.456557+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:26.456557+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196850,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4665,"num_write_kb":37316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4977","reported_seq":5058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:27.276415+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:27.276415+0000","last_peered":"2024-09-15T03:08:27.276415+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:27.276415+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40745785,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":39798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2872}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4740","reported_seq":4803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.522703+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:26.522703+0000","last_peered":"2024-09-15T03:08:26.522703+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:26.522703+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38794866,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4738,"num_write_kb":37893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4912","reported_seq":4968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.355020+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:26.355020+0000","last_peered":"2024-09-15T03:08:26.355020+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:26.355020+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40212082,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4893","reported_seq":4937,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.232848+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:26.232848+0000","last_peered":"2024-09-15T03:08:26.232848+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:26.232848+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4893,"log_dups_size":0,"ondisk_log_size":4893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40083456,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4893,"num_write_kb":39144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4880","reported_seq":4944,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.576776+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:26.576776+0000","last_peered":"2024-09-15T03:08:26.576776+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:26.576776+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39925362,"num_objects":4874,"num_object_clones":0,"num_object_copies":19496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4878,"num_write_kb":39004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2822}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4941","reported_seq":5004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.468660+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:26.468660+0000","last_peered":"2024-09-15T03:08:26.468660+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:26.468660+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40433266,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":39500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4855","reported_seq":4912,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.398686+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:26.398686+0000","last_peered":"2024-09-15T03:08:26.398686+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:26.398686+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39701732,"num_objects":4847,"num_object_clones":0,"num_object_copies":19388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4864","reported_seq":4947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:27.340754+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:27.340754+0000","last_peered":"2024-09-15T03:08:27.340754+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:27.340754+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39827058,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4868","reported_seq":4926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.378787+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:26.378787+0000","last_peered":"2024-09-15T03:08:26.378787+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:26.378787+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4868,"log_dups_size":0,"ondisk_log_size":4868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39834027,"num_objects":4863,"num_object_clones":0,"num_object_copies":19452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":38908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4830","reported_seq":7086,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.468002+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:26.468002+0000","last_peered":"2024-09-15T03:08:26.468002+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:26.468002+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39557945,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3946}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4932","reported_seq":5009,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:27.298212+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:27.298212+0000","last_peered":"2024-09-15T03:08:27.298212+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:27.298212+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40393529,"num_objects":4931,"num_object_clones":0,"num_object_copies":19724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":39447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2892}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4760","reported_seq":4810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.407816+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:26.407816+0000","last_peered":"2024-09-15T03:08:26.407816+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:26.407816+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958706,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4758,"num_write_kb":38060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2720}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4823","reported_seq":4904,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:27.346327+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:27.346327+0000","last_peered":"2024-09-15T03:08:27.346327+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:27.346327+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4823,"log_dups_size":0,"ondisk_log_size":4823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482994,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":38565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4891","reported_seq":4954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.462048+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:26.462048+0000","last_peered":"2024-09-15T03:08:26.462048+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:26.462048+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40023666,"num_objects":4886,"num_object_clones":0,"num_object_copies":19544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":39100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2804}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635853372,"num_objects":77623,"num_object_clones":0,"num_object_copies":310492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77650,"num_write_kb":621082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1143173120,"data_stored":1143173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77677,"ondisk_log_size":77677,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611952,"kb_used_data":309928,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93759888,"statfs":{"total":96636764160,"available":96010125312,"internally_reserved":0,"allocated":317366272,"data_stored":317207294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,6,8,11,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.32900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.314}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.28499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620496,"kb_used_data":310664,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93751344,"statfs":{"total":96636764160,"available":96001376256,"internally_reserved":0,"allocated":318119936,"data_stored":317964104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.0059999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.24399999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.316}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":1.7749999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490496,"kb_used_data":184312,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93881344,"statfs":{"total":96636764160,"available":96134496256,"internally_reserved":0,"allocated":188735488,"data_stored":188576510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,10,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":1.8620000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.24199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":1.899},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.44400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620784,"kb_used_data":310504,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93751056,"statfs":{"total":96636764160,"available":96001081344,"internally_reserved":0,"allocated":317956096,"data_stored":317797118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.247},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":14.301}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":6.3630000000000004},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317706240,"data_stored":317706240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189333504,"data_stored":189333504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318541824,"data_stored":318541824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":317591552,"data_stored":317591552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:31.608 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:32.398 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 176 20 79150 79130 3.51149 2.125 0.0662618 0.0444848 2024-09-15T03:08:32.639 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:32.680 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:32.743 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-15T03:08:30.711511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635853372,"num_objects":77623,"num_object_clones":0,"num_object_copies":310492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77650,"num_write_kb":621082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77677,"ondisk_log_size":77677,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2343728,"kb_used_data":1115408,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375143632,"statfs":{"total":386547056640,"available":384147079168,"internally_reserved":0,"allocated":1142177792,"data_stored":1141545026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,5,8,12,21,11,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22364160,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.012220"},"pg_stats":[{"pgid":"1.f","version":"31'4844","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.318801+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:26.318801+0000","last_peered":"2024-09-15T03:08:26.318801+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:26.318801+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39682048,"num_objects":4844,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4667","reported_seq":4729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.456557+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:26.456557+0000","last_peered":"2024-09-15T03:08:26.456557+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:26.456557+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38196850,"num_objects":4663,"num_object_clones":0,"num_object_copies":18652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4665,"num_write_kb":37316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4977","reported_seq":5058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:27.276415+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:27.276415+0000","last_peered":"2024-09-15T03:08:27.276415+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:27.276415+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40745785,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":39798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2872}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4740","reported_seq":4803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.522703+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:26.522703+0000","last_peered":"2024-09-15T03:08:26.522703+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:26.522703+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38794866,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4738,"num_write_kb":37893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4912","reported_seq":4968,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.355020+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:26.355020+0000","last_peered":"2024-09-15T03:08:26.355020+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:26.355020+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4912,"log_dups_size":0,"ondisk_log_size":4912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40212082,"num_objects":4909,"num_object_clones":0,"num_object_copies":19636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4893","reported_seq":4937,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.232848+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:26.232848+0000","last_peered":"2024-09-15T03:08:26.232848+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:26.232848+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4893,"log_dups_size":0,"ondisk_log_size":4893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40083456,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4893,"num_write_kb":39144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4880","reported_seq":4944,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.576776+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:26.576776+0000","last_peered":"2024-09-15T03:08:26.576776+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:26.576776+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39925362,"num_objects":4874,"num_object_clones":0,"num_object_copies":19496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4878,"num_write_kb":39004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2822}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4941","reported_seq":5004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.468660+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:26.468660+0000","last_peered":"2024-09-15T03:08:26.468660+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:26.468660+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40433266,"num_objects":4936,"num_object_clones":0,"num_object_copies":19744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4939,"num_write_kb":39500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2855}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4855","reported_seq":4912,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.398686+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:26.398686+0000","last_peered":"2024-09-15T03:08:26.398686+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:26.398686+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39701732,"num_objects":4847,"num_object_clones":0,"num_object_copies":19388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4852,"num_write_kb":38793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4864","reported_seq":4947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:27.340754+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:27.340754+0000","last_peered":"2024-09-15T03:08:27.340754+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:27.340754+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39827058,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4862,"num_write_kb":38894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4868","reported_seq":4926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.378787+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:26.378787+0000","last_peered":"2024-09-15T03:08:26.378787+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:26.378787+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4868,"log_dups_size":0,"ondisk_log_size":4868,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39834027,"num_objects":4863,"num_object_clones":0,"num_object_copies":19452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4865,"num_write_kb":38908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4830","reported_seq":7086,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.468002+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:26.468002+0000","last_peered":"2024-09-15T03:08:26.468002+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:26.468002+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4830,"log_dups_size":0,"ondisk_log_size":4830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39557945,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":883,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3946}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4932","reported_seq":5009,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:27.298212+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:27.298212+0000","last_peered":"2024-09-15T03:08:27.298212+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:27.298212+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4932,"log_dups_size":0,"ondisk_log_size":4932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40393529,"num_objects":4931,"num_object_clones":0,"num_object_copies":19724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":39447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2892}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4760","reported_seq":4810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.407816+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:26.407816+0000","last_peered":"2024-09-15T03:08:26.407816+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:26.407816+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38958706,"num_objects":4756,"num_object_clones":0,"num_object_copies":19024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4758,"num_write_kb":38060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2720}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4823","reported_seq":4904,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:27.346327+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:27.346327+0000","last_peered":"2024-09-15T03:08:27.346327+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:27.346327+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4823,"log_dups_size":0,"ondisk_log_size":4823,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482994,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4821,"num_write_kb":38565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2832}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4891","reported_seq":4954,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:26.462048+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:26.462048+0000","last_peered":"2024-09-15T03:08:26.462048+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:26.462048+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40023666,"num_objects":4886,"num_object_clones":0,"num_object_copies":19544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":39100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2804}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":635853372,"num_objects":77623,"num_object_clones":0,"num_object_copies":310492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31547,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77650,"num_write_kb":621082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9084928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1143173120,"data_stored":1143173120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77677,"ondisk_log_size":77677,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611952,"kb_used_data":309928,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93759888,"statfs":{"total":96636764160,"available":96010125312,"internally_reserved":0,"allocated":317366272,"data_stored":317207294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,5,6,8,11,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.32900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.314}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":0.28499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620496,"kb_used_data":310664,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93751344,"statfs":{"total":96636764160,"available":96001376256,"internally_reserved":0,"allocated":318119936,"data_stored":317964104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.0059999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.24399999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.316}]},{"osd":3,"last update":"Sun Sep 15 03:07:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5430000000000001,"5min":1.1919999999999999,"15min":0.80000000000000004},"min":{"1min":0.25800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":21.600999999999999,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.5310000000000001,"5min":1.121,"15min":0.88600000000000001},"min":{"1min":0.26900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":19.274000000000001,"5min":19.274000000000001,"15min":19.274000000000001},"last":1.7749999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490496,"kb_used_data":184312,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93881344,"statfs":{"total":96636764160,"available":96134496256,"internally_reserved":0,"allocated":188735488,"data_stored":188576510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,10,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":1.8620000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.24199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":1.899},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.44400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":620784,"kb_used_data":310504,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93751056,"statfs":{"total":96636764160,"available":96001081344,"internally_reserved":0,"allocated":317956096,"data_stored":317797118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.247},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":14.301}]},{"osd":2,"last update":"Sun Sep 15 03:07:24 2024","interfaces":[{"interface":"back","average":{"1min":2.7109999999999999,"5min":1.29,"15min":1.0529999999999999},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":12.170999999999999,"5min":12.170999999999999,"15min":12.170999999999999},"last":6.3630000000000004},{"interface":"front","average":{"1min":2.0720000000000001,"5min":0.90200000000000002,"15min":0.70699999999999996},"min":{"1min":0.23499999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":8.1159999999999997,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317706240,"data_stored":317706240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189333504,"data_stored":189333504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318541824,"data_stored":318541824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":317591552,"data_stored":317591552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:32.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:08:32.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4844", 'reported_seq': 4886, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.318801+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:08:26.318801+0000', 'last_peered': '2024-09-15T03:08:26.318801+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:08:26.318801+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4844, 'log_dups_size': 0, 'ondisk_log_size': 4844, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39682048, 'num_objects': 4844, 'num_object_clones': 0, 'num_object_copies': 19376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4844, 'num_write_kb': 38752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2868}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:32.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:08:32.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4667", 'reported_seq': 4729, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.456557+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:08:26.456557+0000', 'last_peered': '2024-09-15T03:08:26.456557+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:08:26.456557+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4667, 'log_dups_size': 0, 'ondisk_log_size': 4667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38196850, 'num_objects': 4663, 'num_object_clones': 0, 'num_object_copies': 18652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4665, 'num_write_kb': 37316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2665}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:32.744 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:08:32.744 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4977", 'reported_seq': 5058, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:27.276415+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:08:27.276415+0000', 'last_peered': '2024-09-15T03:08:27.276415+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:08:27.276415+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4977, 'log_dups_size': 0, 'ondisk_log_size': 4977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40745785, 'num_objects': 4974, 'num_object_clones': 0, 'num_object_copies': 19896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4976, 'num_write_kb': 39798, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2872}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4740", 'reported_seq': 4803, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.522703+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:08:26.522703+0000', 'last_peered': '2024-09-15T03:08:26.522703+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:08:26.522703+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4740, 'log_dups_size': 0, 'ondisk_log_size': 4740, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38794866, 'num_objects': 4736, 'num_object_clones': 0, 'num_object_copies': 18944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4738, 'num_write_kb': 37893, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2793}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4912", 'reported_seq': 4968, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.355020+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:08:26.355020+0000', 'last_peered': '2024-09-15T03:08:26.355020+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:08:26.355020+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4912, 'log_dups_size': 0, 'ondisk_log_size': 4912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40212082, 'num_objects': 4909, 'num_object_clones': 0, 'num_object_copies': 19636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4910, 'num_write_kb': 39277, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2802}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4893", 'reported_seq': 4937, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.232848+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:08:26.232848+0000', 'last_peered': '2024-09-15T03:08:26.232848+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:08:26.232848+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4893, 'log_dups_size': 0, 'ondisk_log_size': 4893, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40083456, 'num_objects': 4893, 'num_object_clones': 0, 'num_object_copies': 19572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4893, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4893, 'num_write_kb': 39144, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2837}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4880", 'reported_seq': 4944, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.576776+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:08:26.576776+0000', 'last_peered': '2024-09-15T03:08:26.576776+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:08:26.576776+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4880, 'log_dups_size': 0, 'ondisk_log_size': 4880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39925362, 'num_objects': 4874, 'num_object_clones': 0, 'num_object_copies': 19496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4878, 'num_write_kb': 39004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2822}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4941", 'reported_seq': 5004, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.468660+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:08:26.468660+0000', 'last_peered': '2024-09-15T03:08:26.468660+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:08:26.468660+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4941, 'log_dups_size': 0, 'ondisk_log_size': 4941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40433266, 'num_objects': 4936, 'num_object_clones': 0, 'num_object_copies': 19744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4939, 'num_write_kb': 39500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:32.745 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4855", 'reported_seq': 4912, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.398686+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:08:26.398686+0000', 'last_peered': '2024-09-15T03:08:26.398686+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:08:26.398686+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4855, 'log_dups_size': 0, 'ondisk_log_size': 4855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39701732, 'num_objects': 4847, 'num_object_clones': 0, 'num_object_copies': 19388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4852, 'num_write_kb': 38793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2805}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4864", 'reported_seq': 4947, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:27.340754+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:08:27.340754+0000', 'last_peered': '2024-09-15T03:08:27.340754+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:08:27.340754+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4864, 'log_dups_size': 0, 'ondisk_log_size': 4864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39827058, 'num_objects': 4862, 'num_object_clones': 0, 'num_object_copies': 19448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4862, 'num_write_kb': 38894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2804}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4868", 'reported_seq': 4926, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.378787+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:08:26.378787+0000', 'last_peered': '2024-09-15T03:08:26.378787+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:08:26.378787+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4868, 'log_dups_size': 0, 'ondisk_log_size': 4868, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39834027, 'num_objects': 4863, 'num_object_clones': 0, 'num_object_copies': 19452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4865, 'num_write_kb': 38908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2785}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4830", 'reported_seq': 7086, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.468002+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:08:26.468002+0000', 'last_peered': '2024-09-15T03:08:26.468002+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:08:26.468002+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4830, 'log_dups_size': 0, 'ondisk_log_size': 4830, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39557945, 'num_objects': 4829, 'num_object_clones': 0, 'num_object_copies': 19316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 883, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4829, 'num_write_kb': 38631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1110, 'num_bytes_recovered': 9084928, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3946}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4932", 'reported_seq': 5009, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:27.298212+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:08:27.298212+0000', 'last_peered': '2024-09-15T03:08:27.298212+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:08:27.298212+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4932, 'log_dups_size': 0, 'ondisk_log_size': 4932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40393529, 'num_objects': 4931, 'num_object_clones': 0, 'num_object_copies': 19724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4931, 'num_write_kb': 39447, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2892}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:08:32.746 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4760", 'reported_seq': 4810, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.407816+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:08:26.407816+0000', 'last_peered': '2024-09-15T03:08:26.407816+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:08:26.407816+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4760, 'log_dups_size': 0, 'ondisk_log_size': 4760, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38958706, 'num_objects': 4756, 'num_object_clones': 0, 'num_object_copies': 19024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4758, 'num_write_kb': 38060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2720}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:32.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:08:32.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4823", 'reported_seq': 4904, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:27.346327+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:08:27.346327+0000', 'last_peered': '2024-09-15T03:08:27.346327+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:08:27.346327+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4823, 'log_dups_size': 0, 'ondisk_log_size': 4823, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39482994, 'num_objects': 4820, 'num_object_clones': 0, 'num_object_copies': 19280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4821, 'num_write_kb': 38565, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:32.747 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:08:32.747 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4891", 'reported_seq': 4954, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:26.462048+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:08:26.462048+0000', 'last_peered': '2024-09-15T03:08:26.462048+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:08:26.462048+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4891, 'log_dups_size': 0, 'ondisk_log_size': 4891, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40023666, 'num_objects': 4886, 'num_object_clones': 0, 'num_object_copies': 19544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4889, 'num_write_kb': 39100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2804}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:32.747 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:08:33.398 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 177 20 79408 79388 3.50304 2.01562 0.0569048 0.0445873 2024-09-15T03:08:34.243 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:34.243 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31492/316000 objects degraded (9.966%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":237,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79000,"data_bytes":647133756,"bytes_used":2408685568,"bytes_avail":384138371072,"bytes_total":386547056640,"degraded_objects":31492,"degraded_total":316000,"degraded_ratio":0.099658227848101272,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":102301,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"write_bytes_sec":2801009,"read_op_per_sec":0,"write_op_per_sec":341},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:08:34.291 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:08:34.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:34.401 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 178 20 79668 79648 3.4947 2.03125 0.181813 0.044692 2024-09-15T03:08:35.403 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 179 20 79931 79911 3.48663 2.05469 0.112703 0.0447954 2024-09-15T03:08:35.844 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:35.852 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:35.870 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-15T03:08:34.712532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":647133756,"num_objects":79000,"num_object_clones":0,"num_object_copies":316000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79027,"num_write_kb":632098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1191,"num_bytes_recovered":9535488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79054,"ondisk_log_size":79054,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2352232,"kb_used_data":1136392,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375135128,"statfs":{"total":386547056640,"available":384138371072,"internally_reserved":0,"allocated":1163665408,"data_stored":1163032642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,4,15,10,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22364160,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2730,"num_write_kb":21840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011640"},"pg_stats":[{"pgid":"1.f","version":"31'4920","reported_seq":4962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.474725+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:31.474725+0000","last_peered":"2024-09-15T03:08:31.474725+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:31.474725+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4920,"log_dups_size":0,"ondisk_log_size":4920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40304640,"num_objects":4920,"num_object_clones":0,"num_object_copies":19680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4920,"num_write_kb":39360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4739","reported_seq":4801,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.514974+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:31.514974+0000","last_peered":"2024-09-15T03:08:31.514974+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:31.514974+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786674,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5055","reported_seq":5136,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:32.326631+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:32.326631+0000","last_peered":"2024-09-15T03:08:32.326631+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:32.326631+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5055,"log_dups_size":0,"ondisk_log_size":5055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41384761,"num_objects":5052,"num_object_clones":0,"num_object_copies":20208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5054,"num_write_kb":40422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2950}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4826","reported_seq":4889,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.579585+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:31.579585+0000","last_peered":"2024-09-15T03:08:31.579585+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:31.579585+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39499378,"num_objects":4822,"num_object_clones":0,"num_object_copies":19288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":38581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5009","reported_seq":5064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.335889+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:31.335889+0000","last_peered":"2024-09-15T03:08:31.335889+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:31.335889+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41006706,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":40053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4984","reported_seq":5028,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.381597+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:31.381597+0000","last_peered":"2024-09-15T03:08:31.381597+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:31.381597+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40828928,"num_objects":4984,"num_object_clones":0,"num_object_copies":19936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4984,"num_write_kb":39872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2928}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4971","reported_seq":5036,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.373462+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:31.373462+0000","last_peered":"2024-09-15T03:08:31.373462+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:31.373462+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40670834,"num_objects":4965,"num_object_clones":0,"num_object_copies":19860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4969,"num_write_kb":39732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2913}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5032","reported_seq":5095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.579812+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:31.579812+0000","last_peered":"2024-09-15T03:08:31.579812+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:31.579812+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5032,"log_dups_size":0,"ondisk_log_size":5032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41178738,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4940","reported_seq":4995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.480070+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:31.480070+0000","last_peered":"2024-09-15T03:08:31.480070+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:31.480070+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4940,"log_dups_size":0,"ondisk_log_size":4940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40398052,"num_objects":4932,"num_object_clones":0,"num_object_copies":19728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4937,"num_write_kb":39473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4951","reported_seq":5031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:32.341730+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:32.341730+0000","last_peered":"2024-09-15T03:08:32.341730+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:32.341730+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40539762,"num_objects":4949,"num_object_clones":0,"num_object_copies":19796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4949,"num_write_kb":39590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2891}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4940","reported_seq":4998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.510189+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:31.510189+0000","last_peered":"2024-09-15T03:08:31.510189+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:31.510189+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4940,"log_dups_size":0,"ondisk_log_size":4940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40423851,"num_objects":4935,"num_object_clones":0,"num_object_copies":19740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4937,"num_write_kb":39484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4919","reported_seq":7285,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.524041+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:31.524041+0000","last_peered":"2024-09-15T03:08:31.524041+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:31.524041+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40287033,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1165,"num_bytes_recovered":9535488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4090}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5008","reported_seq":5085,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:32.251528+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:32.251528+0000","last_peered":"2024-09-15T03:08:32.251528+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:32.251528+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41016121,"num_objects":5007,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":40055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2968}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4853","reported_seq":4903,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.486248+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:31.486248+0000","last_peered":"2024-09-15T03:08:31.486248+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:31.486248+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4853,"log_dups_size":0,"ondisk_log_size":4853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39720562,"num_objects":4849,"num_object_clones":0,"num_object_copies":19396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4851,"num_write_kb":38804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4923","reported_seq":5004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:32.282865+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:32.282865+0000","last_peered":"2024-09-15T03:08:32.282865+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:32.282865+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40302194,"num_objects":4920,"num_object_clones":0,"num_object_copies":19680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4921,"num_write_kb":39365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4984","reported_seq":5047,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:31.590794+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:31.590794+0000","last_peered":"2024-09-15T03:08:31.590794+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:31.590794+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4984,"log_dups_size":0,"ondisk_log_size":4984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785522,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4982,"num_write_kb":39844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2897}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":647133756,"num_objects":79000,"num_object_clones":0,"num_object_copies":316000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79027,"num_write_kb":632098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1191,"num_bytes_recovered":9535488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1165946880,"data_stored":1165946880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79054,"ondisk_log_size":79054,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621580,"kb_used_data":315396,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93750260,"statfs":{"total":96636764160,"available":96000266240,"internally_reserved":0,"allocated":322965504,"data_stored":322806526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5,"5min":1.855,"15min":1.7470000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":17.408999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.8360000000000001,"5min":2.02,"15min":2.0510000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":17.542000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.45900000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":7.6070000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":2.3879999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:07:31 2024","interfaces":[{"interface":"back","average":{"1min":2.2469999999999999,"5min":0.875,"15min":0.64600000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.909000000000001,"5min":14.909000000000001,"15min":14.909000000000001},"last":7.7329999999999997},{"interface":"front","average":{"1min":1.399,"5min":1.028,"15min":0.96599999999999997},"min":{"1min":0.251,"5min":0.188,"15min":0.188},"max":{"1min":6.1609999999999996,"5min":6.1609999999999996,"15min":6.1609999999999996},"last":4.7629999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617452,"kb_used_data":315940,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93754388,"statfs":{"total":96636764160,"available":96004493312,"internally_reserved":0,"allocated":323522560,"data_stored":323366728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":3.6179999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":3.577}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":6.9870000000000001},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.248}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.314},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.35099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500388,"kb_used_data":190044,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93871452,"statfs":{"total":96636764160,"available":96124366848,"internally_reserved":0,"allocated":194605056,"data_stored":194446078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,5,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.746},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.25600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.77800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.434}]}]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":612812,"kb_used_data":315012,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93759028,"statfs":{"total":96636764160,"available":96009244672,"internally_reserved":0,"allocated":322572288,"data_stored":322413310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.373}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":1.133}]},{"osd":3,"last update":"Sun Sep 15 03:07:30 2024","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.1100000000000001,"15min":0.81000000000000005},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":14.882,"5min":14.882,"15min":14.882},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.784,"5min":0.73199999999999998,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.6,"5min":11.6,"15min":11.6},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":323325952,"data_stored":323325952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195215360,"data_stored":195215360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":324165632,"data_stored":324165632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323239936,"data_stored":323239936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:36.403 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:08:36.402803+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0448927 2024-09-15T03:08:36.403 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:08:36.403 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 180 20 80203 80183 3.47906 2.125 0.0840955 0.0448927 2024-09-15T03:08:37.404 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 181 20 80478 80458 3.4717 2.14844 0.0705339 0.0449904 2024-09-15T03:08:38.405 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 182 20 80754 80734 3.46445 2.15625 0.0600041 0.045085 2024-09-15T03:08:38.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:39.408 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 183 20 81029 81009 3.45726 2.14844 0.0260046 0.0451807 2024-09-15T03:08:40.407 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 184 20 81297 81277 3.44983 2.09375 0.0176714 0.0452754 2024-09-15T03:08:40.614 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:40.698 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:40.735 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-15T03:08:38.714896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658094652,"num_objects":80338,"num_object_clones":0,"num_object_copies":321352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80365,"num_write_kb":642802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":9953280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80392,"ondisk_log_size":80392,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2377672,"kb_used_data":1157672,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375109688,"statfs":{"total":386547056640,"available":384112320512,"internally_reserved":0,"allocated":1185456128,"data_stored":1184823362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,1,7,10,7,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3049,"num_write_kb":24392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010560"},"pg_stats":[{"pgid":"1.f","version":"31'4998","reported_seq":5040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.579915+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:36.579915+0000","last_peered":"2024-09-15T03:08:36.579915+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:36.579915+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40943616,"num_objects":4998,"num_object_clones":0,"num_object_copies":19992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4998,"num_write_kb":39984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4822","reported_seq":4884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.644866+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:36.644866+0000","last_peered":"2024-09-15T03:08:36.644866+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:36.644866+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39466610,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4820,"num_write_kb":38556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5139","reported_seq":5220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:37.331235+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:37.331235+0000","last_peered":"2024-09-15T03:08:37.331235+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:37.331235+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42072889,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":41094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3034}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4909","reported_seq":4972,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.458293+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:36.458293+0000","last_peered":"2024-09-15T03:08:36.458293+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:36.458293+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4909,"log_dups_size":0,"ondisk_log_size":4909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40179314,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":39245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5110","reported_seq":5166,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.541714+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:36.541714+0000","last_peered":"2024-09-15T03:08:36.541714+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:36.541714+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41834098,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":40861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5076","reported_seq":5120,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.490692+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:36.490692+0000","last_peered":"2024-09-15T03:08:36.490692+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:36.490692+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41582592,"num_objects":5076,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":40608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5052","reported_seq":5117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.618773+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:36.618773+0000","last_peered":"2024-09-15T03:08:36.618773+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:36.618773+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41334386,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":40380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2994}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5125","reported_seq":5188,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.535501+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:36.535501+0000","last_peered":"2024-09-15T03:08:36.535501+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:36.535501+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5125,"log_dups_size":0,"ondisk_log_size":5125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41940594,"num_objects":5120,"num_object_clones":0,"num_object_copies":20480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5123,"num_write_kb":40972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5014","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.522022+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:36.522022+0000","last_peered":"2024-09-15T03:08:36.522022+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:36.522022+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41004260,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2964}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5030","reported_seq":5113,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:37.354852+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:37.354852+0000","last_peered":"2024-09-15T03:08:37.354852+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:37.354852+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41186930,"num_objects":5028,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5028,"num_write_kb":40222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5008","reported_seq":5065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.584640+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:36.584640+0000","last_peered":"2024-09-15T03:08:36.584640+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:36.584640+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40980907,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5009","reported_seq":7477,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.509323+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:36.509323+0000","last_peered":"2024-09-15T03:08:36.509323+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:36.509323+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41024313,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1216,"num_bytes_recovered":9953280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4231}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5107","reported_seq":5184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:37.352285+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:37.352285+0000","last_peered":"2024-09-15T03:08:37.352285+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:37.352285+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41827129,"num_objects":5106,"num_object_clones":0,"num_object_copies":20424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5106,"num_write_kb":40847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4916","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.532852+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:36.532852+0000","last_peered":"2024-09-15T03:08:36.532852+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:36.532852+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236658,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4914,"num_write_kb":39308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5006","reported_seq":5085,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:37.331246+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:37.331246+0000","last_peered":"2024-09-15T03:08:37.331246+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:37.331246+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40982130,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3015}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5071","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.576065+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:36.576065+0000","last_peered":"2024-09-15T03:08:36.576065+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:36.576065+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41498226,"num_objects":5066,"num_object_clones":0,"num_object_copies":20264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":40540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":658094652,"num_objects":80338,"num_object_clones":0,"num_object_copies":321352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80365,"num_write_kb":642802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":9953280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1188040704,"data_stored":1188040704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80392,"ondisk_log_size":80392,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631040,"kb_used_data":320696,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93740800,"statfs":{"total":96636764160,"available":95990579200,"internally_reserved":0,"allocated":328392704,"data_stored":328233726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,3,3,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":7.9020000000000001},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":2.1469999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":7.9779999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.316},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":7.9450000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626904,"kb_used_data":321232,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93744936,"statfs":{"total":96636764160,"available":95994814464,"internally_reserved":0,"allocated":328941568,"data_stored":328785736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.6100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":1.577},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.27600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.45700000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497508,"kb_used_data":195484,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93874332,"statfs":{"total":96636764160,"available":96127315968,"internally_reserved":0,"allocated":200175616,"data_stored":200016638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.188},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.55900000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.373},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.46700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.40600000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622220,"kb_used_data":320260,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93749620,"statfs":{"total":96636764160,"available":95999610880,"internally_reserved":0,"allocated":327946240,"data_stored":327787262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.5179999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.29199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":328765440,"data_stored":328765440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200884224,"data_stored":200884224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329637888,"data_stored":329637888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328753152,"data_stored":328753152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:40.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:41.407 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 185 20 81557 81537 3.44216 2.03125 0.0880281 0.0453815 2024-09-15T03:08:42.037 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:42.064 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:42.090 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-15T03:08:40.715609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658094652,"num_objects":80338,"num_object_clones":0,"num_object_copies":321352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80365,"num_write_kb":642802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":9953280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80392,"ondisk_log_size":80392,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2377672,"kb_used_data":1157672,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375109688,"statfs":{"total":386547056640,"available":384112320512,"internally_reserved":0,"allocated":1185456128,"data_stored":1184823362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,1,7,10,7,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22241280,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2715,"num_write_kb":21720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010703"},"pg_stats":[{"pgid":"1.f","version":"31'4998","reported_seq":5040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.579915+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:36.579915+0000","last_peered":"2024-09-15T03:08:36.579915+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:36.579915+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40943616,"num_objects":4998,"num_object_clones":0,"num_object_copies":19992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4998,"num_write_kb":39984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3022}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4822","reported_seq":4884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.644866+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:36.644866+0000","last_peered":"2024-09-15T03:08:36.644866+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:36.644866+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4822,"log_dups_size":0,"ondisk_log_size":4822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39466610,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4820,"num_write_kb":38556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5139","reported_seq":5220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:37.331235+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:37.331235+0000","last_peered":"2024-09-15T03:08:37.331235+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:37.331235+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42072889,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5138,"num_write_kb":41094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3034}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4909","reported_seq":4972,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.458293+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:36.458293+0000","last_peered":"2024-09-15T03:08:36.458293+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:36.458293+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4909,"log_dups_size":0,"ondisk_log_size":4909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40179314,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":39245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5110","reported_seq":5166,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.541714+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:36.541714+0000","last_peered":"2024-09-15T03:08:36.541714+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:36.541714+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5110,"log_dups_size":0,"ondisk_log_size":5110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41834098,"num_objects":5107,"num_object_clones":0,"num_object_copies":20428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5108,"num_write_kb":40861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3000}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5076","reported_seq":5120,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.490692+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:36.490692+0000","last_peered":"2024-09-15T03:08:36.490692+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:36.490692+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41582592,"num_objects":5076,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":40608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5052","reported_seq":5117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.618773+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:36.618773+0000","last_peered":"2024-09-15T03:08:36.618773+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:36.618773+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41334386,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":40380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2994}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5125","reported_seq":5188,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.535501+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:36.535501+0000","last_peered":"2024-09-15T03:08:36.535501+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:36.535501+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5125,"log_dups_size":0,"ondisk_log_size":5125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41940594,"num_objects":5120,"num_object_clones":0,"num_object_copies":20480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5123,"num_write_kb":40972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3039}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5014","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.522022+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:36.522022+0000","last_peered":"2024-09-15T03:08:36.522022+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:36.522022+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41004260,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2964}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5030","reported_seq":5113,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:37.354852+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:37.354852+0000","last_peered":"2024-09-15T03:08:37.354852+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:37.354852+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41186930,"num_objects":5028,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5028,"num_write_kb":40222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2970}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5008","reported_seq":5065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.584640+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:36.584640+0000","last_peered":"2024-09-15T03:08:36.584640+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:36.584640+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40980907,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5009","reported_seq":7477,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.509323+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:36.509323+0000","last_peered":"2024-09-15T03:08:36.509323+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:36.509323+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41024313,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1216,"num_bytes_recovered":9953280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4231}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5107","reported_seq":5184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:37.352285+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:37.352285+0000","last_peered":"2024-09-15T03:08:37.352285+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:37.352285+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5107,"log_dups_size":0,"ondisk_log_size":5107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41827129,"num_objects":5106,"num_object_clones":0,"num_object_copies":20424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5106,"num_write_kb":40847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4916","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.532852+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:36.532852+0000","last_peered":"2024-09-15T03:08:36.532852+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:36.532852+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236658,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4914,"num_write_kb":39308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5006","reported_seq":5085,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:37.331246+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:37.331246+0000","last_peered":"2024-09-15T03:08:37.331246+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:37.331246+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40982130,"num_objects":5003,"num_object_clones":0,"num_object_copies":20012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":40029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3015}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5071","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:36.576065+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:36.576065+0000","last_peered":"2024-09-15T03:08:36.576065+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:36.576065+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41498226,"num_objects":5066,"num_object_clones":0,"num_object_copies":20264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":40540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2984}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":658094652,"num_objects":80338,"num_object_clones":0,"num_object_copies":321352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80365,"num_write_kb":642802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1242,"num_bytes_recovered":9953280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1188040704,"data_stored":1188040704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80392,"ondisk_log_size":80392,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631040,"kb_used_data":320696,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93740800,"statfs":{"total":96636764160,"available":95990579200,"internally_reserved":0,"allocated":328392704,"data_stored":328233726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,3,3,5,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":7.9020000000000001},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":2.1469999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":7.9779999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.316},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":7.9450000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626904,"kb_used_data":321232,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93744936,"statfs":{"total":96636764160,"available":95994814464,"internally_reserved":0,"allocated":328941568,"data_stored":328785736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.6100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":1.577},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.27600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.45700000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497508,"kb_used_data":195484,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93874332,"statfs":{"total":96636764160,"available":96127315968,"internally_reserved":0,"allocated":200175616,"data_stored":200016638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.188},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.55900000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.373},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.46700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.40600000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622220,"kb_used_data":320260,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93749620,"statfs":{"total":96636764160,"available":95999610880,"internally_reserved":0,"allocated":327946240,"data_stored":327787262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.5179999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.42599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.29199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.318}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":328765440,"data_stored":328765440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200884224,"data_stored":200884224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329637888,"data_stored":329637888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328753152,"data_stored":328753152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:42.092 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:08:42.092 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4998", 'reported_seq': 5040, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.579915+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:08:36.579915+0000', 'last_peered': '2024-09-15T03:08:36.579915+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:08:36.579915+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4998, 'log_dups_size': 0, 'ondisk_log_size': 4998, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40943616, 'num_objects': 4998, 'num_object_clones': 0, 'num_object_copies': 19992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4998, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4998, 'num_write_kb': 39984, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3022}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:42.092 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:08:42.092 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4822", 'reported_seq': 4884, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.644866+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:08:36.644866+0000', 'last_peered': '2024-09-15T03:08:36.644866+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:08:36.644866+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4822, 'log_dups_size': 0, 'ondisk_log_size': 4822, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39466610, 'num_objects': 4818, 'num_object_clones': 0, 'num_object_copies': 19272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4820, 'num_write_kb': 38556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2820}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:42.092 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:08:42.092 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5139", 'reported_seq': 5220, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:37.331235+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:08:37.331235+0000', 'last_peered': '2024-09-15T03:08:37.331235+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:08:37.331235+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5139, 'log_dups_size': 0, 'ondisk_log_size': 5139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42072889, 'num_objects': 5136, 'num_object_clones': 0, 'num_object_copies': 20544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5138, 'num_write_kb': 41094, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3034}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:42.092 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:08:42.093 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4909", 'reported_seq': 4972, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.458293+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:08:36.458293+0000', 'last_peered': '2024-09-15T03:08:36.458293+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:08:36.458293+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4909, 'log_dups_size': 0, 'ondisk_log_size': 4909, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40179314, 'num_objects': 4905, 'num_object_clones': 0, 'num_object_copies': 19620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4907, 'num_write_kb': 39245, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:42.093 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:08:42.093 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5110", 'reported_seq': 5166, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.541714+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:08:36.541714+0000', 'last_peered': '2024-09-15T03:08:36.541714+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:08:36.541714+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5110, 'log_dups_size': 0, 'ondisk_log_size': 5110, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41834098, 'num_objects': 5107, 'num_object_clones': 0, 'num_object_copies': 20428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5108, 'num_write_kb': 40861, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3000}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:42.093 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:08:42.093 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5076", 'reported_seq': 5120, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.490692+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:08:36.490692+0000', 'last_peered': '2024-09-15T03:08:36.490692+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:08:36.490692+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5076, 'log_dups_size': 0, 'ondisk_log_size': 5076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41582592, 'num_objects': 5076, 'num_object_clones': 0, 'num_object_copies': 20304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5076, 'num_write_kb': 40608, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3020}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:42.093 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:08:42.093 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5052", 'reported_seq': 5117, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.618773+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:08:36.618773+0000', 'last_peered': '2024-09-15T03:08:36.618773+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:08:36.618773+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5052, 'log_dups_size': 0, 'ondisk_log_size': 5052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41334386, 'num_objects': 5046, 'num_object_clones': 0, 'num_object_copies': 20184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5050, 'num_write_kb': 40380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2994}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:42.093 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:08:42.094 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5125", 'reported_seq': 5188, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.535501+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:08:36.535501+0000', 'last_peered': '2024-09-15T03:08:36.535501+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:08:36.535501+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5125, 'log_dups_size': 0, 'ondisk_log_size': 5125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41940594, 'num_objects': 5120, 'num_object_clones': 0, 'num_object_copies': 20480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5123, 'num_write_kb': 40972, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3039}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:42.094 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:08:42.094 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5014", 'reported_seq': 5071, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.522022+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:08:36.522022+0000', 'last_peered': '2024-09-15T03:08:36.522022+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:08:36.522022+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5014, 'log_dups_size': 0, 'ondisk_log_size': 5014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41004260, 'num_objects': 5006, 'num_object_clones': 0, 'num_object_copies': 20024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5011, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2964}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:42.094 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:08:42.094 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5030", 'reported_seq': 5113, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:37.354852+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:08:37.354852+0000', 'last_peered': '2024-09-15T03:08:37.354852+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:08:37.354852+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5030, 'log_dups_size': 0, 'ondisk_log_size': 5030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41186930, 'num_objects': 5028, 'num_object_clones': 0, 'num_object_copies': 20112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5028, 'num_write_kb': 40222, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2970}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:42.094 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:08:42.094 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5008", 'reported_seq': 5065, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.584640+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:08:36.584640+0000', 'last_peered': '2024-09-15T03:08:36.584640+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:08:36.584640+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5008, 'log_dups_size': 0, 'ondisk_log_size': 5008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40980907, 'num_objects': 5003, 'num_object_clones': 0, 'num_object_copies': 20012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5005, 'num_write_kb': 40028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2925}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:42.094 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5009", 'reported_seq': 7477, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.509323+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:08:36.509323+0000', 'last_peered': '2024-09-15T03:08:36.509323+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:08:36.509323+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5009, 'log_dups_size': 0, 'ondisk_log_size': 5009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41024313, 'num_objects': 5008, 'num_object_clones': 0, 'num_object_copies': 20032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 777, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5008, 'num_write_kb': 40063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1216, 'num_bytes_recovered': 9953280, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5107", 'reported_seq': 5184, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:37.352285+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:08:37.352285+0000', 'last_peered': '2024-09-15T03:08:37.352285+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:08:37.352285+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5107, 'log_dups_size': 0, 'ondisk_log_size': 5107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41827129, 'num_objects': 5106, 'num_object_clones': 0, 'num_object_copies': 20424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5106, 'num_write_kb': 40847, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3067}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4916", 'reported_seq': 4966, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.532852+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:08:36.532852+0000', 'last_peered': '2024-09-15T03:08:36.532852+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:08:36.532852+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4916, 'log_dups_size': 0, 'ondisk_log_size': 4916, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40236658, 'num_objects': 4912, 'num_object_clones': 0, 'num_object_copies': 19648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4914, 'num_write_kb': 39308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5006", 'reported_seq': 5085, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:37.331246+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:08:37.331246+0000', 'last_peered': '2024-09-15T03:08:37.331246+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:08:37.331246+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5006, 'log_dups_size': 0, 'ondisk_log_size': 5006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40982130, 'num_objects': 5003, 'num_object_clones': 0, 'num_object_copies': 20012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5004, 'num_write_kb': 40029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3015}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:08:42.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5071", 'reported_seq': 5134, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:36.576065+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:08:36.576065+0000', 'last_peered': '2024-09-15T03:08:36.576065+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:08:36.576065+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5071, 'log_dups_size': 0, 'ondisk_log_size': 5071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41498226, 'num_objects': 5066, 'num_object_clones': 0, 'num_object_copies': 20264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5069, 'num_write_kb': 40540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2984}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:42.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:08:42.407 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 186 20 81811 81791 3.43432 1.98438 0.107871 0.0454711 2024-09-15T03:08:43.408 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 187 20 82075 82055 3.42697 2.0625 0.122649 0.0455753 2024-09-15T03:08:44.011 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:44.012 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31375/326736 objects degraded (9.603%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":247,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81684,"data_bytes":669121084,"bytes_used":2465873920,"bytes_avail":384081182720,"bytes_total":386547056640,"degraded_objects":31375,"degraded_total":326736,"degraded_ratio":0.096025537436952163,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":117343,"recovering_keys_per_sec":0,"num_objects_recovered":173,"num_bytes_recovered":1409024,"num_keys_recovered":0,"write_bytes_sec":2770528,"read_op_per_sec":0,"write_op_per_sec":338},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:08:44.050 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:08:44.051 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:44.408 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 188 20 82338 82318 3.41967 2.05469 0.111947 0.0456801 2024-09-15T03:08:45.266 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:45.357 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-15T03:08:44.719733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669121084,"num_objects":81684,"num_object_clones":0,"num_object_copies":326736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81711,"num_write_kb":653570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":10493952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81738,"ondisk_log_size":81738,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2408080,"kb_used_data":1179756,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375079280,"statfs":{"total":386547056640,"available":384081182720,"internally_reserved":0,"allocated":1208070144,"data_stored":1207437378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,1,4,7,6,13,16,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21987328,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2684,"num_write_kb":21472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007753"},"pg_stats":[{"pgid":"1.f","version":"31'5074","reported_seq":5113,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.649823+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:41.649823+0000","last_peered":"2024-09-15T03:08:41.649823+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:41.649823+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5074,"log_dups_size":0,"ondisk_log_size":5074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41566208,"num_objects":5074,"num_object_clones":0,"num_object_copies":20296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5074,"num_write_kb":40592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4904","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.597239+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:41.597239+0000","last_peered":"2024-09-15T03:08:41.597239+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:41.597239+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40138354,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4902,"num_write_kb":39212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2902}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5220","reported_seq":5301,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:42.219769+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:42.219769+0000","last_peered":"2024-09-15T03:08:42.219769+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:42.219769+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42736441,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5219,"num_write_kb":41742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3115}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5002","reported_seq":5065,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.566006+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:41.566006+0000","last_peered":"2024-09-15T03:08:41.566006+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:41.566006+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40941170,"num_objects":4998,"num_object_clones":0,"num_object_copies":19992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5000,"num_write_kb":39989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5185","reported_seq":5241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.433779+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:41.433779+0000","last_peered":"2024-09-15T03:08:41.433779+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:41.433779+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42448498,"num_objects":5182,"num_object_clones":0,"num_object_copies":20728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":41461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5164","reported_seq":5208,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.578764+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:41.578764+0000","last_peered":"2024-09-15T03:08:41.578764+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:41.578764+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5164,"log_dups_size":0,"ondisk_log_size":5164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42303488,"num_objects":5164,"num_object_clones":0,"num_object_copies":20656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5164,"num_write_kb":41312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5147","reported_seq":5212,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.478382+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:41.478382+0000","last_peered":"2024-09-15T03:08:41.478382+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:41.478382+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42112626,"num_objects":5141,"num_object_clones":0,"num_object_copies":20564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5145,"num_write_kb":41140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5205","reported_seq":5268,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.521922+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:41.521922+0000","last_peered":"2024-09-15T03:08:41.521922+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:41.521922+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5205,"log_dups_size":0,"ondisk_log_size":5205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595954,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3119}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5099","reported_seq":5156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.404784+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:41.404784+0000","last_peered":"2024-09-15T03:08:41.404784+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:41.404784+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41700580,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5096,"num_write_kb":40745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3049}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5125","reported_seq":5208,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:42.370555+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:42.370555+0000","last_peered":"2024-09-15T03:08:42.370555+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:42.370555+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5125,"log_dups_size":0,"ondisk_log_size":5125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41965170,"num_objects":5123,"num_object_clones":0,"num_object_copies":20492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5123,"num_write_kb":40982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5088","reported_seq":5146,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.533539+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:41.533539+0000","last_peered":"2024-09-15T03:08:41.533539+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:41.533539+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41636267,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5087","reported_seq":7689,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.651933+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:41.651933+0000","last_peered":"2024-09-15T03:08:41.651933+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:41.651933+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41663289,"num_objects":5086,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":40687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":10493952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4375}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5172","reported_seq":5249,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:42.252188+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:42.252188+0000","last_peered":"2024-09-15T03:08:42.252188+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:42.252188+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5172,"log_dups_size":0,"ondisk_log_size":5172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42359609,"num_objects":5171,"num_object_clones":0,"num_object_copies":20684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":41367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3132}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5015","reported_seq":5064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.638738+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:41.638738+0000","last_peered":"2024-09-15T03:08:41.638738+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:41.638738+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047666,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5013,"num_write_kb":40100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2975}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5089","reported_seq":5170,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:42.425380+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:42.425380+0000","last_peered":"2024-09-15T03:08:42.425380+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:42.425380+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41662066,"num_objects":5086,"num_object_clones":0,"num_object_copies":20344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5087,"num_write_kb":40693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3098}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5162","reported_seq":5225,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:41.590841+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:41.590841+0000","last_peered":"2024-09-15T03:08:41.590841+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:41.590841+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42243698,"num_objects":5157,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5160,"num_write_kb":41268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":669121084,"num_objects":81684,"num_object_clones":0,"num_object_copies":326736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81711,"num_write_kb":653570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1309,"num_bytes_recovered":10493952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1210511360,"data_stored":1210511360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81738,"ondisk_log_size":81738,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628148,"kb_used_data":326124,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93743692,"statfs":{"total":96636764160,"available":95993540608,"internally_reserved":0,"allocated":333950976,"data_stored":333791998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,1,2,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.26100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.28899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.39800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636548,"kb_used_data":326712,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93735292,"statfs":{"total":96636764160,"available":95984939008,"internally_reserved":0,"allocated":334553088,"data_stored":334397256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.37},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.56000000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.50800000000000001},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.53200000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.33800000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507368,"kb_used_data":201184,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93864472,"statfs":{"total":96636764160,"available":96117219328,"internally_reserved":0,"allocated":206012416,"data_stored":205853438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.26400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.312},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.376}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.35599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636016,"kb_used_data":325736,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93735824,"statfs":{"total":96636764160,"available":95985483776,"internally_reserved":0,"allocated":333553664,"data_stored":333394686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,2,3,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":6.3419999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":6.2889999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":6.4909999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":6.4589999999999996},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":6.4189999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":334331904,"data_stored":334331904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206696448,"data_stored":206696448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335130624,"data_stored":335130624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334352384,"data_stored":334352384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:45.408 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 189 20 82588 82568 3.41191 1.95312 0.191982 0.0457758 2024-09-15T03:08:46.408 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 190 20 82846 82826 3.40457 2.01562 0.0575819 0.045878 2024-09-15T03:08:47.408 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 191 20 83109 83089 3.3975 2.05469 0.0449713 0.0459641 2024-09-15T03:08:48.360 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:48.408 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 192 20 83381 83361 3.39087 2.125 0.035445 0.046063 2024-09-15T03:08:49.408 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 193 20 83629 83609 3.38334 1.9375 0.0431694 0.0461633 2024-09-15T03:08:49.742 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:49.754 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:49.784 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-15T03:08:48.721169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":679893564,"num_objects":82999,"num_object_clones":0,"num_object_copies":331996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83026,"num_write_kb":664090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":10797056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83053,"ondisk_log_size":83053,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2416360,"kb_used_data":1200520,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375071000,"statfs":{"total":386547056640,"available":384072704000,"internally_reserved":0,"allocated":1229332480,"data_stored":1228699714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,5,6,11,16,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24625152,"num_objects":3006,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3006,"num_write_kb":24048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006812"},"pg_stats":[{"pgid":"1.f","version":"31'5165","reported_seq":5207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.668732+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:46.668732+0000","last_peered":"2024-09-15T03:08:46.668732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:46.668732+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42311680,"num_objects":5165,"num_object_clones":0,"num_object_copies":20660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5165,"num_write_kb":41320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4968","reported_seq":5029,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.507931+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:46.507931+0000","last_peered":"2024-09-15T03:08:46.507931+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:46.507931+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40662642,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4966,"num_write_kb":39724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5299","reported_seq":5380,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:47.437805+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:47.437805+0000","last_peered":"2024-09-15T03:08:47.437805+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:47.437805+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5299,"log_dups_size":0,"ondisk_log_size":5299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43383609,"num_objects":5296,"num_object_clones":0,"num_object_copies":21184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5298,"num_write_kb":42374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5084","reported_seq":5146,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.597746+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:46.597746+0000","last_peered":"2024-09-15T03:08:46.597746+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:46.597746+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41612914,"num_objects":5080,"num_object_clones":0,"num_object_copies":20320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5082,"num_write_kb":40645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5260","reported_seq":5316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.346772+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:46.346772+0000","last_peered":"2024-09-15T03:08:46.346772+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:46.346772+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5260,"log_dups_size":0,"ondisk_log_size":5260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43062898,"num_objects":5257,"num_object_clones":0,"num_object_copies":21028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5258,"num_write_kb":42061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3150}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5253","reported_seq":5296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.660872+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:46.660872+0000","last_peered":"2024-09-15T03:08:46.660872+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:46.660872+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43032576,"num_objects":5253,"num_object_clones":0,"num_object_copies":21012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5253,"num_write_kb":42024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5220","reported_seq":5285,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.625755+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:46.625755+0000","last_peered":"2024-09-15T03:08:46.625755+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:46.625755+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710642,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5282","reported_seq":5343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.648945+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:46.648945+0000","last_peered":"2024-09-15T03:08:46.648945+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:46.648945+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226738,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5280,"num_write_kb":42228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3196}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5192","reported_seq":5249,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.602376+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:46.602376+0000","last_peered":"2024-09-15T03:08:46.602376+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:46.602376+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5192,"log_dups_size":0,"ondisk_log_size":5192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42462436,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":41489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5207","reported_seq":5290,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:47.338761+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:47.338761+0000","last_peered":"2024-09-15T03:08:47.338761+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:47.338761+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42636914,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3147}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5170","reported_seq":5228,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.665794+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:46.665794+0000","last_peered":"2024-09-15T03:08:46.665794+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:46.665794+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42308011,"num_objects":5165,"num_object_clones":0,"num_object_copies":20660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":41324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5179","reported_seq":7854,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.691286+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:46.691286+0000","last_peered":"2024-09-15T03:08:46.691286+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:46.691286+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42416953,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5178,"num_write_kb":41423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":10797056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5256","reported_seq":5333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:47.404963+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:47.404963+0000","last_peered":"2024-09-15T03:08:47.404963+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:47.404963+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43047737,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":42039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5102","reported_seq":5152,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.626518+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:46.626518+0000","last_peered":"2024-09-15T03:08:46.626518+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:46.626518+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41760370,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5100,"num_write_kb":40796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5175","reported_seq":5255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:47.399258+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:47.399258+0000","last_peered":"2024-09-15T03:08:47.399258+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:47.399258+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42366578,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":41381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3184}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5241","reported_seq":5304,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.546444+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:46.546444+0000","last_peered":"2024-09-15T03:08:46.546444+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:46.546444+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42890866,"num_objects":5236,"num_object_clones":0,"num_object_copies":20944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":41900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":679893564,"num_objects":82999,"num_object_clones":0,"num_object_copies":331996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83026,"num_write_kb":664090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":10797056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1231888384,"data_stored":1231888384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83053,"ondisk_log_size":83053,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637528,"kb_used_data":331344,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93734312,"statfs":{"total":96636764160,"available":95983935488,"internally_reserved":0,"allocated":339296256,"data_stored":339137278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,5,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":4.2140000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.91600000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.79800000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633384,"kb_used_data":331872,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93738456,"statfs":{"total":96636764160,"available":95988178944,"internally_reserved":0,"allocated":339836928,"data_stored":339681096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":3.0779999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.40200000000000002},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.34300000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.32000000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516724,"kb_used_data":206380,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93855116,"statfs":{"total":96636764160,"available":96107638784,"internally_reserved":0,"allocated":211333120,"data_stored":211174142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.318},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.378}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.34699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.44500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628724,"kb_used_data":330924,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93743116,"statfs":{"total":96636764160,"available":95992950784,"internally_reserved":0,"allocated":338866176,"data_stored":338707198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.33900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.37},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.48199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339611648,"data_stored":339611648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212148224,"data_stored":212148224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340471808,"data_stored":340471808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339656704,"data_stored":339656704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:49.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:50.409 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 194 20 83875 83855 3.37581 1.92188 0.0524665 0.0462685 2024-09-15T03:08:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:51.290 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:51.332 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-15T03:08:50.721705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":679893564,"num_objects":82999,"num_object_clones":0,"num_object_copies":331996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83026,"num_write_kb":664090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":10797056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83053,"ondisk_log_size":83053,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2416360,"kb_used_data":1200520,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375071000,"statfs":{"total":386547056640,"available":384072704000,"internally_reserved":0,"allocated":1229332480,"data_stored":1228699714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,5,6,11,16,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21798912,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2661,"num_write_kb":21288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006809"},"pg_stats":[{"pgid":"1.f","version":"31'5165","reported_seq":5207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.668732+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:46.668732+0000","last_peered":"2024-09-15T03:08:46.668732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:46.668732+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42311680,"num_objects":5165,"num_object_clones":0,"num_object_copies":20660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5165,"num_write_kb":41320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4968","reported_seq":5029,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.507931+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:46.507931+0000","last_peered":"2024-09-15T03:08:46.507931+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:46.507931+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":4968,"log_dups_size":0,"ondisk_log_size":4968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40662642,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4966,"num_write_kb":39724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5299","reported_seq":5380,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:47.437805+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:47.437805+0000","last_peered":"2024-09-15T03:08:47.437805+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:47.437805+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5299,"log_dups_size":0,"ondisk_log_size":5299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43383609,"num_objects":5296,"num_object_clones":0,"num_object_copies":21184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5298,"num_write_kb":42374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5084","reported_seq":5146,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.597746+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:46.597746+0000","last_peered":"2024-09-15T03:08:46.597746+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:46.597746+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41612914,"num_objects":5080,"num_object_clones":0,"num_object_copies":20320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5082,"num_write_kb":40645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5260","reported_seq":5316,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.346772+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:46.346772+0000","last_peered":"2024-09-15T03:08:46.346772+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:46.346772+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5260,"log_dups_size":0,"ondisk_log_size":5260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43062898,"num_objects":5257,"num_object_clones":0,"num_object_copies":21028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5258,"num_write_kb":42061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3150}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5253","reported_seq":5296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.660872+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:46.660872+0000","last_peered":"2024-09-15T03:08:46.660872+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:46.660872+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5253,"log_dups_size":0,"ondisk_log_size":5253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43032576,"num_objects":5253,"num_object_clones":0,"num_object_copies":21012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5253,"num_write_kb":42024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5220","reported_seq":5285,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.625755+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:46.625755+0000","last_peered":"2024-09-15T03:08:46.625755+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:46.625755+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5220,"log_dups_size":0,"ondisk_log_size":5220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710642,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5218,"num_write_kb":41724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5282","reported_seq":5343,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.648945+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:46.648945+0000","last_peered":"2024-09-15T03:08:46.648945+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:46.648945+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226738,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5280,"num_write_kb":42228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3196}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5192","reported_seq":5249,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.602376+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:46.602376+0000","last_peered":"2024-09-15T03:08:46.602376+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:46.602376+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5192,"log_dups_size":0,"ondisk_log_size":5192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42462436,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5189,"num_write_kb":41489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5207","reported_seq":5290,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:47.338761+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:47.338761+0000","last_peered":"2024-09-15T03:08:47.338761+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:47.338761+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42636914,"num_objects":5205,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":41638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3147}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5170","reported_seq":5228,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.665794+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:46.665794+0000","last_peered":"2024-09-15T03:08:46.665794+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:46.665794+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5170,"log_dups_size":0,"ondisk_log_size":5170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42308011,"num_objects":5165,"num_object_clones":0,"num_object_copies":20660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":41324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5179","reported_seq":7854,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.691286+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:46.691286+0000","last_peered":"2024-09-15T03:08:46.691286+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:46.691286+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42416953,"num_objects":5178,"num_object_clones":0,"num_object_copies":20712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5178,"num_write_kb":41423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1320,"num_bytes_recovered":10797056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5256","reported_seq":5333,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:47.404963+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:47.404963+0000","last_peered":"2024-09-15T03:08:47.404963+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:47.404963+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43047737,"num_objects":5255,"num_object_clones":0,"num_object_copies":21020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":42039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5102","reported_seq":5152,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.626518+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:46.626518+0000","last_peered":"2024-09-15T03:08:46.626518+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:46.626518+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41760370,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5100,"num_write_kb":40796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5175","reported_seq":5255,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:47.399258+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:47.399258+0000","last_peered":"2024-09-15T03:08:47.399258+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:47.399258+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42366578,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":41381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3184}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5241","reported_seq":5304,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:46.546444+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:46.546444+0000","last_peered":"2024-09-15T03:08:46.546444+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:46.546444+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42890866,"num_objects":5236,"num_object_clones":0,"num_object_copies":20944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5239,"num_write_kb":41900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":679893564,"num_objects":82999,"num_object_clones":0,"num_object_copies":331996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83026,"num_write_kb":664090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1346,"num_bytes_recovered":10797056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1231888384,"data_stored":1231888384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83053,"ondisk_log_size":83053,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637528,"kb_used_data":331344,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93734312,"statfs":{"total":96636764160,"available":95983935488,"internally_reserved":0,"allocated":339296256,"data_stored":339137278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,5,8,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":4.2140000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.91600000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.79800000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":633384,"kb_used_data":331872,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93738456,"statfs":{"total":96636764160,"available":95988178944,"internally_reserved":0,"allocated":339836928,"data_stored":339681096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":3.0779999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.40200000000000002},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.34300000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.32000000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516724,"kb_used_data":206380,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93855116,"statfs":{"total":96636764160,"available":96107638784,"internally_reserved":0,"allocated":211333120,"data_stored":211174142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.318},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.378}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.34699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.44500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628724,"kb_used_data":330924,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93743116,"statfs":{"total":96636764160,"available":95992950784,"internally_reserved":0,"allocated":338866176,"data_stored":338707198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.33900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.37},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.48199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339611648,"data_stored":339611648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212148224,"data_stored":212148224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340471808,"data_stored":340471808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339656704,"data_stored":339656704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:51.333 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:08:51.333 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5165", 'reported_seq': 5207, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.668732+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:08:46.668732+0000', 'last_peered': '2024-09-15T03:08:46.668732+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:08:46.668732+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5165, 'log_dups_size': 0, 'ondisk_log_size': 5165, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42311680, 'num_objects': 5165, 'num_object_clones': 0, 'num_object_copies': 20660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5165, 'num_write_kb': 41320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3189}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:51.333 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:08:51.333 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4968", 'reported_seq': 5029, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.507931+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:08:46.507931+0000', 'last_peered': '2024-09-15T03:08:46.507931+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:08:46.507931+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 4968, 'log_dups_size': 0, 'ondisk_log_size': 4968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40662642, 'num_objects': 4964, 'num_object_clones': 0, 'num_object_copies': 19856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4966, 'num_write_kb': 39724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5299", 'reported_seq': 5380, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:47.437805+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:08:47.437805+0000', 'last_peered': '2024-09-15T03:08:47.437805+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:08:47.437805+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5299, 'log_dups_size': 0, 'ondisk_log_size': 5299, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43383609, 'num_objects': 5296, 'num_object_clones': 0, 'num_object_copies': 21184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5298, 'num_write_kb': 42374, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3194}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5084", 'reported_seq': 5146, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.597746+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:08:46.597746+0000', 'last_peered': '2024-09-15T03:08:46.597746+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:08:46.597746+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5084, 'log_dups_size': 0, 'ondisk_log_size': 5084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41612914, 'num_objects': 5080, 'num_object_clones': 0, 'num_object_copies': 20320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5082, 'num_write_kb': 40645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3137}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5260", 'reported_seq': 5316, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.346772+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:08:46.346772+0000', 'last_peered': '2024-09-15T03:08:46.346772+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:08:46.346772+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5260, 'log_dups_size': 0, 'ondisk_log_size': 5260, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43062898, 'num_objects': 5257, 'num_object_clones': 0, 'num_object_copies': 21028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5258, 'num_write_kb': 42061, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3150}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5253", 'reported_seq': 5296, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.660872+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:08:46.660872+0000', 'last_peered': '2024-09-15T03:08:46.660872+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:08:46.660872+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5253, 'log_dups_size': 0, 'ondisk_log_size': 5253, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43032576, 'num_objects': 5253, 'num_object_clones': 0, 'num_object_copies': 21012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5253, 'num_write_kb': 42024, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3197}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:08:51.334 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5220", 'reported_seq': 5285, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.625755+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:08:46.625755+0000', 'last_peered': '2024-09-15T03:08:46.625755+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:08:46.625755+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5220, 'log_dups_size': 0, 'ondisk_log_size': 5220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42710642, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5218, 'num_write_kb': 41724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3162}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5282", 'reported_seq': 5343, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.648945+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:08:46.648945+0000', 'last_peered': '2024-09-15T03:08:46.648945+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:08:46.648945+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5282, 'log_dups_size': 0, 'ondisk_log_size': 5282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43226738, 'num_objects': 5277, 'num_object_clones': 0, 'num_object_copies': 21108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5280, 'num_write_kb': 42228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5192", 'reported_seq': 5249, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.602376+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:08:46.602376+0000', 'last_peered': '2024-09-15T03:08:46.602376+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:08:46.602376+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5192, 'log_dups_size': 0, 'ondisk_log_size': 5192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42462436, 'num_objects': 5184, 'num_object_clones': 0, 'num_object_copies': 20736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5189, 'num_write_kb': 41489, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3142}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5207", 'reported_seq': 5290, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:47.338761+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:08:47.338761+0000', 'last_peered': '2024-09-15T03:08:47.338761+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:08:47.338761+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5207, 'log_dups_size': 0, 'ondisk_log_size': 5207, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42636914, 'num_objects': 5205, 'num_object_clones': 0, 'num_object_copies': 20820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5205, 'num_write_kb': 41638, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3147}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5170", 'reported_seq': 5228, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.665794+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:08:46.665794+0000', 'last_peered': '2024-09-15T03:08:46.665794+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:08:46.665794+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5170, 'log_dups_size': 0, 'ondisk_log_size': 5170, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42308011, 'num_objects': 5165, 'num_object_clones': 0, 'num_object_copies': 20660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5167, 'num_write_kb': 41324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3087}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:08:51.335 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5179", 'reported_seq': 7854, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.691286+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:08:46.691286+0000', 'last_peered': '2024-09-15T03:08:46.691286+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:08:46.691286+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5179, 'log_dups_size': 0, 'ondisk_log_size': 5179, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42416953, 'num_objects': 5178, 'num_object_clones': 0, 'num_object_copies': 20712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 674, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5178, 'num_write_kb': 41423, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1320, 'num_bytes_recovered': 10797056, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:51.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:08:51.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5256", 'reported_seq': 5333, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:47.404963+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:08:47.404963+0000', 'last_peered': '2024-09-15T03:08:47.404963+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:08:47.404963+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5256, 'log_dups_size': 0, 'ondisk_log_size': 5256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43047737, 'num_objects': 5255, 'num_object_clones': 0, 'num_object_copies': 21020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5255, 'num_write_kb': 42039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:51.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:08:51.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5102", 'reported_seq': 5152, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.626518+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:08:46.626518+0000', 'last_peered': '2024-09-15T03:08:46.626518+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:08:46.626518+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5102, 'log_dups_size': 0, 'ondisk_log_size': 5102, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41760370, 'num_objects': 5098, 'num_object_clones': 0, 'num_object_copies': 20392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5100, 'num_write_kb': 40796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3062}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:51.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:08:51.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5175", 'reported_seq': 5255, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:47.399258+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:08:47.399258+0000', 'last_peered': '2024-09-15T03:08:47.399258+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:08:47.399258+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5175, 'log_dups_size': 0, 'ondisk_log_size': 5175, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42366578, 'num_objects': 5172, 'num_object_clones': 0, 'num_object_copies': 20688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5173, 'num_write_kb': 41381, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:51.336 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:08:51.336 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5241", 'reported_seq': 5304, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:46.546444+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:08:46.546444+0000', 'last_peered': '2024-09-15T03:08:46.546444+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:08:46.546444+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5241, 'log_dups_size': 0, 'ondisk_log_size': 5241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42890866, 'num_objects': 5236, 'num_object_clones': 0, 'num_object_copies': 20944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5239, 'num_write_kb': 41900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3154}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:51.336 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:08:51.409 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 195 20 84133 84113 3.36883 2.01562 0.160596 0.0463632 2024-09-15T03:08:52.409 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 196 20 84396 84376 3.36213 2.05469 0.0392891 0.0464556 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31338/331996 objects degraded (9.439%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":255,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":82999,"data_bytes":679893564,"bytes_used":2474352640,"bytes_avail":384072704000,"bytes_total":386547056640,"degraded_objects":31338,"degraded_total":331996,"degraded_ratio":0.094392703526548521,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":70274,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":843776,"num_keys_recovered":0,"write_bytes_sec":1815545,"read_op_per_sec":0,"write_op_per_sec":221},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:08:52.868 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:08:52.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:53.409 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 197 20 84635 84615 3.35454 1.86719 0.0351403 0.0465627 2024-09-15T03:08:53.797 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:53.824 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:53.858 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-15T03:08:52.722297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":690461244,"num_objects":84289,"num_object_clones":0,"num_object_copies":337156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84316,"num_write_kb":674410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":11255808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84343,"ondisk_log_size":84343,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2445640,"kb_used_data":1221480,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375041720,"statfs":{"total":386547056640,"available":384042721280,"internally_reserved":0,"allocated":1250795520,"data_stored":1250162754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,2,5,11,11,15,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32366592,"num_objects":3951,"num_object_clones":0,"num_object_copies":15804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3951,"num_write_kb":31608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":1302528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006688"},"pg_stats":[{"pgid":"1.f","version":"31'5245","reported_seq":5287,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.774869+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:51.774869+0000","last_peered":"2024-09-15T03:08:51.774869+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:51.774869+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42967040,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5245,"num_write_kb":41960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5034","reported_seq":5096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.672483+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:51.672483+0000","last_peered":"2024-09-15T03:08:51.672483+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:51.672483+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41203314,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5032,"num_write_kb":40252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5399","reported_seq":5480,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:52.370893+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:52.370893+0000","last_peered":"2024-09-15T03:08:52.370893+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:52.370893+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5399,"log_dups_size":0,"ondisk_log_size":5399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44202809,"num_objects":5396,"num_object_clones":0,"num_object_copies":21584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5398,"num_write_kb":43174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5159","reported_seq":5220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.628171+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:51.628171+0000","last_peered":"2024-09-15T03:08:51.628171+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:51.628171+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42227314,"num_objects":5155,"num_object_clones":0,"num_object_copies":20620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":41245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5342","reported_seq":5398,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.527883+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:51.527883+0000","last_peered":"2024-09-15T03:08:51.527883+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:51.527883+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43734642,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5340,"num_write_kb":42717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5333","reported_seq":5377,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.723425+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:51.723425+0000","last_peered":"2024-09-15T03:08:51.723425+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:51.723425+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5333,"log_dups_size":0,"ondisk_log_size":5333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43687936,"num_objects":5333,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5333,"num_write_kb":42664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5293","reported_seq":5358,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.609427+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:51.609427+0000","last_peered":"2024-09-15T03:08:51.609427+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:51.609427+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5293,"log_dups_size":0,"ondisk_log_size":5293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43308658,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":42308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3235}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5379","reported_seq":5442,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.641815+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:51.641815+0000","last_peered":"2024-09-15T03:08:51.641815+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:51.641815+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5379,"log_dups_size":0,"ondisk_log_size":5379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44021362,"num_objects":5374,"num_object_clones":0,"num_object_copies":21496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5377,"num_write_kb":43004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5266","reported_seq":5323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.448196+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:51.448196+0000","last_peered":"2024-09-15T03:08:51.448196+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:51.448196+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43068644,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5284","reported_seq":5367,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:52.499749+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:52.499749+0000","last_peered":"2024-09-15T03:08:52.499749+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:52.499749+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5284,"log_dups_size":0,"ondisk_log_size":5284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43267698,"num_objects":5282,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5282,"num_write_kb":42254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5239","reported_seq":5297,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.469788+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:51.469788+0000","last_peered":"2024-09-15T03:08:51.469788+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:51.469788+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42873259,"num_objects":5234,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":41876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5264","reported_seq":8052,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.783098+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:51.783098+0000","last_peered":"2024-09-15T03:08:51.783098+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:51.783098+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43113273,"num_objects":5263,"num_object_clones":0,"num_object_copies":21052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11255808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5331","reported_seq":5408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:52.399681+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:52.399681+0000","last_peered":"2024-09-15T03:08:52.399681+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:52.399681+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43662137,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5330,"num_write_kb":42639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3291}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5181","reported_seq":5231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.764400+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:51.764400+0000","last_peered":"2024-09-15T03:08:51.764400+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:51.764400+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42407538,"num_objects":5177,"num_object_clones":0,"num_object_copies":20708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5179,"num_write_kb":41428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5280","reported_seq":5361,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:52.482953+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:52.482953+0000","last_peered":"2024-09-15T03:08:52.482953+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:52.482953+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226738,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5278,"num_write_kb":42221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5314","reported_seq":5377,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.559940+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:51.559940+0000","last_peered":"2024-09-15T03:08:51.559940+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:51.559940+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43488882,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":690461244,"num_objects":84289,"num_object_clones":0,"num_object_copies":337156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84316,"num_write_kb":674410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":11255808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253392384,"data_stored":1253392384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84343,"ondisk_log_size":84343,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651092,"kb_used_data":336588,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93720748,"statfs":{"total":96636764160,"available":95970045952,"internally_reserved":0,"allocated":344666112,"data_stored":344507134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":8.5259999999999998},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.214}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":8.6679999999999993}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":8.6349999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642688,"kb_used_data":337016,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93729152,"statfs":{"total":96636764160,"available":95978651648,"internally_reserved":0,"allocated":345104384,"data_stored":344948552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":3.0779999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.39100000000000001},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":3.5209999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":3.2269999999999999},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513856,"kb_used_data":211832,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93857984,"statfs":{"total":96636764160,"available":96110575616,"internally_reserved":0,"allocated":216915968,"data_stored":216756990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.246},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":4.1280000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.496}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.72599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638004,"kb_used_data":336044,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93733836,"statfs":{"total":96636764160,"available":95983448064,"internally_reserved":0,"allocated":344109056,"data_stored":343950078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,0,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":3.5630000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":3.351},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.25600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.314},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344915968,"data_stored":344915968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217698304,"data_stored":217698304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":345763840,"data_stored":345763840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345014272,"data_stored":345014272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:54.413 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 198 20 84891 84871 3.34764 2 0.0185169 0.0466599 2024-09-15T03:08:55.415 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 199 20 85149 85129 3.3409 2.01562 0.0387091 0.0467454 2024-09-15T03:08:56.415 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:08:56.414898+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0468486 2024-09-15T03:08:56.415 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:08:56.416 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 200 20 85406 85386 3.33424 2.00781 0.154984 0.0468486 2024-09-15T03:08:56.860 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:57.415 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 201 20 85639 85619 3.32671 1.82031 0.044091 0.0469441 2024-09-15T03:08:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:58.328 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-15T03:08:56.723323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":690461244,"num_objects":84289,"num_object_clones":0,"num_object_copies":337156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84316,"num_write_kb":674410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":11255808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84343,"ondisk_log_size":84343,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2445640,"kb_used_data":1221480,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375041720,"statfs":{"total":386547056640,"available":384042721280,"internally_reserved":0,"allocated":1250795520,"data_stored":1250162754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,2,5,11,11,15,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21340160,"num_objects":2605,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2605,"num_write_kb":20840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003589"},"pg_stats":[{"pgid":"1.f","version":"31'5245","reported_seq":5287,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.774869+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:51.774869+0000","last_peered":"2024-09-15T03:08:51.774869+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:51.774869+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42967040,"num_objects":5245,"num_object_clones":0,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5245,"num_write_kb":41960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5034","reported_seq":5096,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.672483+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:51.672483+0000","last_peered":"2024-09-15T03:08:51.672483+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:51.672483+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41203314,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5032,"num_write_kb":40252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5399","reported_seq":5480,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:52.370893+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:52.370893+0000","last_peered":"2024-09-15T03:08:52.370893+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:52.370893+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5399,"log_dups_size":0,"ondisk_log_size":5399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44202809,"num_objects":5396,"num_object_clones":0,"num_object_copies":21584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5398,"num_write_kb":43174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5159","reported_seq":5220,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.628171+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:51.628171+0000","last_peered":"2024-09-15T03:08:51.628171+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:51.628171+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42227314,"num_objects":5155,"num_object_clones":0,"num_object_copies":20620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":41245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5342","reported_seq":5398,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.527883+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:51.527883+0000","last_peered":"2024-09-15T03:08:51.527883+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:51.527883+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43734642,"num_objects":5339,"num_object_clones":0,"num_object_copies":21356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5340,"num_write_kb":42717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5333","reported_seq":5377,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.723425+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:51.723425+0000","last_peered":"2024-09-15T03:08:51.723425+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:51.723425+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5333,"log_dups_size":0,"ondisk_log_size":5333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43687936,"num_objects":5333,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5333,"num_write_kb":42664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5293","reported_seq":5358,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.609427+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:51.609427+0000","last_peered":"2024-09-15T03:08:51.609427+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:51.609427+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5293,"log_dups_size":0,"ondisk_log_size":5293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43308658,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5291,"num_write_kb":42308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3235}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5379","reported_seq":5442,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.641815+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:51.641815+0000","last_peered":"2024-09-15T03:08:51.641815+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:51.641815+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5379,"log_dups_size":0,"ondisk_log_size":5379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44021362,"num_objects":5374,"num_object_clones":0,"num_object_copies":21496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5377,"num_write_kb":43004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5266","reported_seq":5323,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.448196+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:51.448196+0000","last_peered":"2024-09-15T03:08:51.448196+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:51.448196+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5266,"log_dups_size":0,"ondisk_log_size":5266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43068644,"num_objects":5258,"num_object_clones":0,"num_object_copies":21032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3216}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5284","reported_seq":5367,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:52.499749+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:52.499749+0000","last_peered":"2024-09-15T03:08:52.499749+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:52.499749+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5284,"log_dups_size":0,"ondisk_log_size":5284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43267698,"num_objects":5282,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5282,"num_write_kb":42254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3224}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5239","reported_seq":5297,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.469788+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:51.469788+0000","last_peered":"2024-09-15T03:08:51.469788+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:51.469788+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42873259,"num_objects":5234,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5236,"num_write_kb":41876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3156}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5264","reported_seq":8052,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.783098+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:51.783098+0000","last_peered":"2024-09-15T03:08:51.783098+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:51.783098+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43113273,"num_objects":5263,"num_object_clones":0,"num_object_copies":21052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":42103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1376,"num_bytes_recovered":11255808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4645}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5331","reported_seq":5408,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:52.399681+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:52.399681+0000","last_peered":"2024-09-15T03:08:52.399681+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:52.399681+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43662137,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5330,"num_write_kb":42639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3291}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5181","reported_seq":5231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.764400+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:51.764400+0000","last_peered":"2024-09-15T03:08:51.764400+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:51.764400+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42407538,"num_objects":5177,"num_object_clones":0,"num_object_copies":20708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5179,"num_write_kb":41428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3141}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5280","reported_seq":5361,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:52.482953+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:52.482953+0000","last_peered":"2024-09-15T03:08:52.482953+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:52.482953+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5280,"log_dups_size":0,"ondisk_log_size":5280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43226738,"num_objects":5277,"num_object_clones":0,"num_object_copies":21108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5278,"num_write_kb":42221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3289}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5314","reported_seq":5377,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:51.559940+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:51.559940+0000","last_peered":"2024-09-15T03:08:51.559940+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:51.559940+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5314,"log_dups_size":0,"ondisk_log_size":5314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43488882,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5312,"num_write_kb":42484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":690461244,"num_objects":84289,"num_object_clones":0,"num_object_copies":337156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84316,"num_write_kb":674410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1402,"num_bytes_recovered":11255808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253392384,"data_stored":1253392384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84343,"ondisk_log_size":84343,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651092,"kb_used_data":336588,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93720748,"statfs":{"total":96636764160,"available":95970045952,"internally_reserved":0,"allocated":344666112,"data_stored":344507134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":8.5259999999999998},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.214}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":8.6679999999999993}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":8.6349999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642688,"kb_used_data":337016,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93729152,"statfs":{"total":96636764160,"available":95978651648,"internally_reserved":0,"allocated":345104384,"data_stored":344948552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":3.0779999999999998},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.39100000000000001},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":3.5209999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":3.2269999999999999},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513856,"kb_used_data":211832,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93857984,"statfs":{"total":96636764160,"available":96110575616,"internally_reserved":0,"allocated":216915968,"data_stored":216756990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.246},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":4.1280000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.496}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.72599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638004,"kb_used_data":336044,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93733836,"statfs":{"total":96636764160,"available":95983448064,"internally_reserved":0,"allocated":344109056,"data_stored":343950078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,0,0,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":3.5630000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":3.351},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.25600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.314},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":344915968,"data_stored":344915968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217698304,"data_stored":217698304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":345763840,"data_stored":345763840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345014272,"data_stored":345014272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:58.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:08:58.418 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 202 20 85897 85877 3.32017 2.01562 0.18764 0.047042 2024-09-15T03:08:59.418 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 203 20 86153 86133 3.31367 2 0.105839 0.0471395 2024-09-15T03:08:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:08:59.853 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:08:59.882 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-15T03:08:58.723785+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":700971580,"num_objects":85572,"num_object_clones":0,"num_object_copies":342288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85599,"num_write_kb":684674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1457,"num_bytes_recovered":11706368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85626,"ondisk_log_size":85626,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2472444,"kb_used_data":1244124,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375014916,"statfs":{"total":386547056640,"available":384015273984,"internally_reserved":0,"allocated":1273982976,"data_stored":1273350210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,6,15,22,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28778496,"num_objects":3513,"num_object_clones":0,"num_object_copies":14052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":28104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003574"},"pg_stats":[{"pgid":"1.f","version":"31'5331","reported_seq":5373,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.793346+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:08:56.793346+0000","last_peered":"2024-09-15T03:08:56.793346+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:08:56.793346+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43671552,"num_objects":5331,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5331,"num_write_kb":42648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3355}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5113","reported_seq":5175,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.734402+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:08:56.734402+0000","last_peered":"2024-09-15T03:08:56.734402+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:08:56.734402+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41850482,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5111,"num_write_kb":40884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3111}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5492","reported_seq":5572,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:57.438140+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:08:57.438140+0000","last_peered":"2024-09-15T03:08:57.438140+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:08:57.438140+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5492,"log_dups_size":0,"ondisk_log_size":5492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44964665,"num_objects":5489,"num_object_clones":0,"num_object_copies":21956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":43918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3387}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5237","reported_seq":5299,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.631931+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:08:56.631931+0000","last_peered":"2024-09-15T03:08:56.631931+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:08:56.631931+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42866290,"num_objects":5233,"num_object_clones":0,"num_object_copies":20932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5235,"num_write_kb":41869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5415","reported_seq":5471,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.789483+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:08:56.789483+0000","last_peered":"2024-09-15T03:08:56.789483+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:08:56.789483+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5415,"log_dups_size":0,"ondisk_log_size":5415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44332658,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5413,"num_write_kb":43301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5397","reported_seq":5441,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.805355+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:08:56.805355+0000","last_peered":"2024-09-15T03:08:56.805355+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:08:56.805355+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44212224,"num_objects":5397,"num_object_clones":0,"num_object_copies":21588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":43176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3341}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5370","reported_seq":5435,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.571942+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:08:56.571942+0000","last_peered":"2024-09-15T03:08:56.571942+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:08:56.571942+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43939442,"num_objects":5364,"num_object_clones":0,"num_object_copies":21456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":42924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3312}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5461","reported_seq":5524,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.596690+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:08:56.596690+0000","last_peered":"2024-09-15T03:08:56.596690+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:08:56.596690+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5461,"log_dups_size":0,"ondisk_log_size":5461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44693106,"num_objects":5456,"num_object_clones":0,"num_object_copies":21824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5459,"num_write_kb":43660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5350","reported_seq":5407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.706882+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:08:56.706882+0000","last_peered":"2024-09-15T03:08:56.706882+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:08:56.706882+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5350,"log_dups_size":0,"ondisk_log_size":5350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43756772,"num_objects":5342,"num_object_clones":0,"num_object_copies":21368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5347,"num_write_kb":42753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5369","reported_seq":5452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:57.420976+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:08:57.420976+0000","last_peered":"2024-09-15T03:08:57.420976+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:08:57.420976+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5369,"log_dups_size":0,"ondisk_log_size":5369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43964018,"num_objects":5367,"num_object_clones":0,"num_object_copies":21468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5367,"num_write_kb":42934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3309}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5326","reported_seq":5384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.691956+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:08:56.691956+0000","last_peered":"2024-09-15T03:08:56.691956+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:08:56.691956+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5326,"log_dups_size":0,"ondisk_log_size":5326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43585963,"num_objects":5321,"num_object_clones":0,"num_object_copies":21284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5323,"num_write_kb":42572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5347","reported_seq":8244,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.739558+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:08:56.739558+0000","last_peered":"2024-09-15T03:08:56.739558+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:08:56.739558+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43793209,"num_objects":5346,"num_object_clones":0,"num_object_copies":21384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":42767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1431,"num_bytes_recovered":11706368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4783}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5428","reported_seq":5505,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:57.397734+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:08:57.397734+0000","last_peered":"2024-09-15T03:08:57.397734+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:08:57.397734+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5428,"log_dups_size":0,"ondisk_log_size":5428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44456761,"num_objects":5427,"num_object_clones":0,"num_object_copies":21708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5427,"num_write_kb":43415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3388}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5242","reported_seq":5292,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.795717+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:08:56.795717+0000","last_peered":"2024-09-15T03:08:56.795717+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:08:56.795717+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42907250,"num_objects":5238,"num_object_clones":0,"num_object_copies":20952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5240,"num_write_kb":41916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3202}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5348","reported_seq":5428,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:57.343023+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:08:57.343023+0000","last_peered":"2024-09-15T03:08:57.343023+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:08:57.343023+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5348,"log_dups_size":0,"ondisk_log_size":5348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43783794,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5346,"num_write_kb":42765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3357}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5400","reported_seq":5463,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:08:56.615850+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:08:56.615850+0000","last_peered":"2024-09-15T03:08:56.615850+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:08:56.615850+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5400,"log_dups_size":0,"ondisk_log_size":5400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44193394,"num_objects":5395,"num_object_clones":0,"num_object_copies":21580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5398,"num_write_kb":43172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3313}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":700971580,"num_objects":85572,"num_object_clones":0,"num_object_copies":342288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85599,"num_write_kb":684674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1457,"num_bytes_recovered":11706368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1274507264,"data_stored":1274507264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85626,"ondisk_log_size":85626,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643728,"kb_used_data":341704,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93728112,"statfs":{"total":96636764160,"available":95977586688,"internally_reserved":0,"allocated":349904896,"data_stored":349745918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,6,12,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.6379999999999999},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.25600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":2.1680000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":1.786}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":2.1989999999999998},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.1520000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651932,"kb_used_data":342100,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93719908,"statfs":{"total":96636764160,"available":95969185792,"internally_reserved":0,"allocated":350310400,"data_stored":350154568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.0669999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":1.097}]}]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":524264,"kb_used_data":218080,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93847576,"statfs":{"total":96636764160,"available":96099917824,"internally_reserved":0,"allocated":223313920,"data_stored":223154942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.247},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.38800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.27600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":0.30599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652520,"kb_used_data":342240,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93719320,"statfs":{"total":96636764160,"available":95968583680,"internally_reserved":0,"allocated":350453760,"data_stored":350294782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.307},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.26300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.372}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":2.6890000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350191616,"data_stored":350191616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":223133696,"data_stored":223133696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":350908416,"data_stored":350908416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":350273536,"data_stored":350273536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:08:59.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:08:59.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5331", 'reported_seq': 5373, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.793346+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:08:56.793346+0000', 'last_peered': '2024-09-15T03:08:56.793346+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:08:56.793346+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5331, 'log_dups_size': 0, 'ondisk_log_size': 5331, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43671552, 'num_objects': 5331, 'num_object_clones': 0, 'num_object_copies': 21324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5331, 'num_write_kb': 42648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3355}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5113", 'reported_seq': 5175, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.734402+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:08:56.734402+0000', 'last_peered': '2024-09-15T03:08:56.734402+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:08:56.734402+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5113, 'log_dups_size': 0, 'ondisk_log_size': 5113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41850482, 'num_objects': 5109, 'num_object_clones': 0, 'num_object_copies': 20436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5111, 'num_write_kb': 40884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5492", 'reported_seq': 5572, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:57.438140+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:08:57.438140+0000', 'last_peered': '2024-09-15T03:08:57.438140+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:08:57.438140+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5492, 'log_dups_size': 0, 'ondisk_log_size': 5492, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44964665, 'num_objects': 5489, 'num_object_clones': 0, 'num_object_copies': 21956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5491, 'num_write_kb': 43918, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3387}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5237", 'reported_seq': 5299, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.631931+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:08:56.631931+0000', 'last_peered': '2024-09-15T03:08:56.631931+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:08:56.631931+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5237, 'log_dups_size': 0, 'ondisk_log_size': 5237, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42866290, 'num_objects': 5233, 'num_object_clones': 0, 'num_object_copies': 20932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5235, 'num_write_kb': 41869, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3290}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5415", 'reported_seq': 5471, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.789483+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:08:56.789483+0000', 'last_peered': '2024-09-15T03:08:56.789483+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:08:56.789483+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5415, 'log_dups_size': 0, 'ondisk_log_size': 5415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44332658, 'num_objects': 5412, 'num_object_clones': 0, 'num_object_copies': 21648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5413, 'num_write_kb': 43301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3305}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:59.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5397", 'reported_seq': 5441, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.805355+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:08:56.805355+0000', 'last_peered': '2024-09-15T03:08:56.805355+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:08:56.805355+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5397, 'log_dups_size': 0, 'ondisk_log_size': 5397, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44212224, 'num_objects': 5397, 'num_object_clones': 0, 'num_object_copies': 21588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5397, 'num_write_kb': 43176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3341}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5370", 'reported_seq': 5435, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.571942+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:08:56.571942+0000', 'last_peered': '2024-09-15T03:08:56.571942+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:08:56.571942+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5370, 'log_dups_size': 0, 'ondisk_log_size': 5370, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43939442, 'num_objects': 5364, 'num_object_clones': 0, 'num_object_copies': 21456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5364, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5368, 'num_write_kb': 42924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3312}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5461", 'reported_seq': 5524, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.596690+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:08:56.596690+0000', 'last_peered': '2024-09-15T03:08:56.596690+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:08:56.596690+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5461, 'log_dups_size': 0, 'ondisk_log_size': 5461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44693106, 'num_objects': 5456, 'num_object_clones': 0, 'num_object_copies': 21824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5459, 'num_write_kb': 43660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5350", 'reported_seq': 5407, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.706882+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:08:56.706882+0000', 'last_peered': '2024-09-15T03:08:56.706882+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:08:56.706882+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5350, 'log_dups_size': 0, 'ondisk_log_size': 5350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43756772, 'num_objects': 5342, 'num_object_clones': 0, 'num_object_copies': 21368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5347, 'num_write_kb': 42753, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3300}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5369", 'reported_seq': 5452, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:57.420976+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:08:57.420976+0000', 'last_peered': '2024-09-15T03:08:57.420976+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:08:57.420976+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5369, 'log_dups_size': 0, 'ondisk_log_size': 5369, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43964018, 'num_objects': 5367, 'num_object_clones': 0, 'num_object_copies': 21468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5367, 'num_write_kb': 42934, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3309}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:08:59.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5326", 'reported_seq': 5384, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.691956+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:08:56.691956+0000', 'last_peered': '2024-09-15T03:08:56.691956+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:08:56.691956+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5326, 'log_dups_size': 0, 'ondisk_log_size': 5326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43585963, 'num_objects': 5321, 'num_object_clones': 0, 'num_object_copies': 21284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5323, 'num_write_kb': 42572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3243}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:59.886 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:08:59.886 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5347", 'reported_seq': 8244, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.739558+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:08:56.739558+0000', 'last_peered': '2024-09-15T03:08:56.739558+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:08:56.739558+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5347, 'log_dups_size': 0, 'ondisk_log_size': 5347, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43793209, 'num_objects': 5346, 'num_object_clones': 0, 'num_object_copies': 21384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 563, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5346, 'num_write_kb': 42767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1431, 'num_bytes_recovered': 11706368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 4783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:59.886 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:08:59.886 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5428", 'reported_seq': 5505, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:57.397734+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:08:57.397734+0000', 'last_peered': '2024-09-15T03:08:57.397734+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:08:57.397734+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5428, 'log_dups_size': 0, 'ondisk_log_size': 5428, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44456761, 'num_objects': 5427, 'num_object_clones': 0, 'num_object_copies': 21708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5427, 'num_write_kb': 43415, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3388}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:59.886 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:08:59.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5242", 'reported_seq': 5292, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.795717+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:08:56.795717+0000', 'last_peered': '2024-09-15T03:08:56.795717+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:08:56.795717+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5242, 'log_dups_size': 0, 'ondisk_log_size': 5242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42907250, 'num_objects': 5238, 'num_object_clones': 0, 'num_object_copies': 20952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5240, 'num_write_kb': 41916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3202}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:08:59.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:08:59.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5348", 'reported_seq': 5428, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:57.343023+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:08:57.343023+0000', 'last_peered': '2024-09-15T03:08:57.343023+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:08:57.343023+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5348, 'log_dups_size': 0, 'ondisk_log_size': 5348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43783794, 'num_objects': 5345, 'num_object_clones': 0, 'num_object_copies': 21380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5346, 'num_write_kb': 42765, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3357}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:08:59.888 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:08:59.888 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5400", 'reported_seq': 5463, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:08:56.615850+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:08:56.615850+0000', 'last_peered': '2024-09-15T03:08:56.615850+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:08:56.615850+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5400, 'log_dups_size': 0, 'ondisk_log_size': 5400, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44193394, 'num_objects': 5395, 'num_object_clones': 0, 'num_object_copies': 21580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5398, 'num_write_kb': 43172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3313}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:08:59.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:09:00.418 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 204 20 86415 86395 3.30746 2.04688 0.0254559 0.0472297 2024-09-15T03:09:01.419 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 205 20 86660 86640 3.30067 1.91406 0.149961 0.0473213 2024-09-15T03:09:01.834 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:01.835 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31227/342288 objects degraded (9.123%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":264,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85572,"data_bytes":700971580,"bytes_used":2531782656,"bytes_avail":384015273984,"bytes_total":386547056640,"degraded_objects":31227,"degraded_total":342288,"degraded_ratio":0.091230192118917405,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":75756,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"write_bytes_sec":1756039,"read_op_per_sec":0,"write_op_per_sec":214},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:09:01.883 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:09:01.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:02.419 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 206 20 86915 86895 3.2943 1.99219 0.0228257 0.0474127 2024-09-15T03:09:03.419 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 207 20 87164 87144 3.28779 1.94531 0.100141 0.0475098 2024-09-15T03:09:03.477 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:03.497 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:03.521 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-15T03:09:02.727122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":711391804,"num_objects":86844,"num_object_clones":0,"num_object_copies":347376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86871,"num_write_kb":694850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":12165120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86898,"ondisk_log_size":86898,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2466752,"kb_used_data":1263392,"kb_used_omap":14,"kb_used_meta":1203313,"kb_avail":375020608,"statfs":{"total":386547056640,"available":384021102592,"internally_reserved":0,"allocated":1293713408,"data_stored":1293080642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232192539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,7,10,6,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31498240,"num_objects":3845,"num_object_clones":0,"num_object_copies":15380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3845,"num_write_kb":30760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005416"},"pg_stats":[{"pgid":"1.f","version":"31'5410","reported_seq":5452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.874929+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:01.874929+0000","last_peered":"2024-09-15T03:09:01.874929+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:01.874929+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44318720,"num_objects":5410,"num_object_clones":0,"num_object_copies":21640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":43280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5200","reported_seq":5260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.735978+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:01.735978+0000","last_peered":"2024-09-15T03:09:01.735978+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:01.735978+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5200,"log_dups_size":0,"ondisk_log_size":5200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42563186,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":41580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5569","reported_seq":5650,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:02.514891+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:02.514891+0000","last_peered":"2024-09-15T03:09:02.514891+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:02.514891+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45595449,"num_objects":5566,"num_object_clones":0,"num_object_copies":22264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5568,"num_write_kb":44534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5313","reported_seq":5376,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.626718+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:01.626718+0000","last_peered":"2024-09-15T03:09:01.626718+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:01.626718+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43488882,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5311,"num_write_kb":42477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5504","reported_seq":5560,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.728985+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:01.728985+0000","last_peered":"2024-09-15T03:09:01.728985+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:01.728985+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5504,"log_dups_size":0,"ondisk_log_size":5504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45061746,"num_objects":5501,"num_object_clones":0,"num_object_copies":22004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5502,"num_write_kb":44013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5470","reported_seq":5514,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.874147+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:01.874147+0000","last_peered":"2024-09-15T03:09:01.874147+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:01.874147+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44810240,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5470,"num_write_kb":43760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5454","reported_seq":5519,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.697543+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:01.697543+0000","last_peered":"2024-09-15T03:09:01.697543+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:01.697543+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44627570,"num_objects":5448,"num_object_clones":0,"num_object_copies":21792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5452,"num_write_kb":43596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5532","reported_seq":5593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.747888+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:01.747888+0000","last_peered":"2024-09-15T03:09:01.747888+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:01.747888+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274738,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":44228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5446","reported_seq":5501,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.895996+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:01.895996+0000","last_peered":"2024-09-15T03:09:01.895996+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:01.895996+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5446,"log_dups_size":0,"ondisk_log_size":5446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44543204,"num_objects":5438,"num_object_clones":0,"num_object_copies":21752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":43521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5451","reported_seq":5534,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:02.420570+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:02.420570+0000","last_peered":"2024-09-15T03:09:02.420570+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:02.420570+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635762,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5449,"num_write_kb":43590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5396","reported_seq":5454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.827041+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:01.827041+0000","last_peered":"2024-09-15T03:09:01.827041+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:01.827041+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44159403,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3313}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5424","reported_seq":8434,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.933590+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:01.933590+0000","last_peered":"2024-09-15T03:09:01.933590+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:01.933590+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44423993,"num_objects":5423,"num_object_clones":0,"num_object_copies":21692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5423,"num_write_kb":43383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":12165120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4916}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5526","reported_seq":5603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:02.505252+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:02.505252+0000","last_peered":"2024-09-15T03:09:02.505252+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:02.505252+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45259577,"num_objects":5525,"num_object_clones":0,"num_object_copies":22100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5525,"num_write_kb":44199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5312","reported_seq":5362,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.707329+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:01.707329+0000","last_peered":"2024-09-15T03:09:01.707329+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:01.707329+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5312,"log_dups_size":0,"ondisk_log_size":5312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43480690,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5310,"num_write_kb":42476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5417","reported_seq":5498,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:02.321054+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:02.321054+0000","last_peered":"2024-09-15T03:09:02.321054+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:02.321054+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44349042,"num_objects":5414,"num_object_clones":0,"num_object_copies":21656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":43317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5474","reported_seq":5537,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.748457+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:01.748457+0000","last_peered":"2024-09-15T03:09:01.748457+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:01.748457+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799602,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5472,"num_write_kb":43764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3387}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":711391804,"num_objects":86844,"num_object_clones":0,"num_object_copies":347376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86871,"num_write_kb":694850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":12165120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1295585280,"data_stored":1295585280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86898,"ondisk_log_size":86898,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652888,"kb_used_data":346704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93718952,"statfs":{"total":96636764160,"available":95968206848,"internally_reserved":0,"allocated":355024896,"data_stored":354865918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":5.6669999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":7.8609999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":4.8220000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":7.9189999999999996},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.7919999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":648624,"kb_used_data":347112,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93723216,"statfs":{"total":96636764160,"available":95972573184,"internally_reserved":0,"allocated":355442688,"data_stored":355286856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":5.8860000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.317},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":5.8390000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521128,"kb_used_data":223264,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93850712,"statfs":{"total":96636764160,"available":96103129088,"internally_reserved":0,"allocated":228622336,"data_stored":228463358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.307},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":3.238}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.219},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":2.355}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.2730000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":4.0549999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644112,"kb_used_data":346312,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93727728,"statfs":{"total":96636764160,"available":95977193472,"internally_reserved":0,"allocated":354623488,"data_stored":354464510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":2.556},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":1.972}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":18.808},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":2.2589999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":1.331},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":1.5549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355360768,"data_stored":355360768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228581376,"data_stored":228581376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356143104,"data_stored":356143104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355500032,"data_stored":355500032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:04.420 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 208 20 87421 87401 3.28163 2.00781 0.0208825 0.0475967 2024-09-15T03:09:05.423 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 209 20 87682 87662 3.27568 2.03906 0.121046 0.0476883 2024-09-15T03:09:06.420 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 210 20 87942 87922 3.26976 2.03125 0.0741593 0.0477731 2024-09-15T03:09:06.524 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:07.420 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 211 20 88169 88149 3.26267 1.77344 0.0446256 0.047873 2024-09-15T03:09:08.148 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:08.161 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:08.186 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-15T03:09:06.728153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":711391804,"num_objects":86844,"num_object_clones":0,"num_object_copies":347376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86871,"num_write_kb":694850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":12165120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86898,"ondisk_log_size":86898,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2466752,"kb_used_data":1263392,"kb_used_omap":14,"kb_used_meta":1203313,"kb_avail":375020608,"statfs":{"total":386547056640,"available":384021102592,"internally_reserved":0,"allocated":1293713408,"data_stored":1293080642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232192539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,5,7,10,6,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20930560,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2555,"num_write_kb":20440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":909312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005317"},"pg_stats":[{"pgid":"1.f","version":"31'5410","reported_seq":5452,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.874929+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:01.874929+0000","last_peered":"2024-09-15T03:09:01.874929+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:01.874929+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44318720,"num_objects":5410,"num_object_clones":0,"num_object_copies":21640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":43280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5200","reported_seq":5260,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.735978+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:01.735978+0000","last_peered":"2024-09-15T03:09:01.735978+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:01.735978+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5200,"log_dups_size":0,"ondisk_log_size":5200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42563186,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":41580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5569","reported_seq":5650,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:02.514891+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:02.514891+0000","last_peered":"2024-09-15T03:09:02.514891+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:02.514891+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45595449,"num_objects":5566,"num_object_clones":0,"num_object_copies":22264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5568,"num_write_kb":44534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5313","reported_seq":5376,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.626718+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:01.626718+0000","last_peered":"2024-09-15T03:09:01.626718+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:01.626718+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5313,"log_dups_size":0,"ondisk_log_size":5313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43488882,"num_objects":5309,"num_object_clones":0,"num_object_copies":21236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5311,"num_write_kb":42477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5504","reported_seq":5560,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.728985+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:01.728985+0000","last_peered":"2024-09-15T03:09:01.728985+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:01.728985+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5504,"log_dups_size":0,"ondisk_log_size":5504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45061746,"num_objects":5501,"num_object_clones":0,"num_object_copies":22004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5502,"num_write_kb":44013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5470","reported_seq":5514,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.874147+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:01.874147+0000","last_peered":"2024-09-15T03:09:01.874147+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:01.874147+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44810240,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5470,"num_write_kb":43760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5454","reported_seq":5519,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.697543+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:01.697543+0000","last_peered":"2024-09-15T03:09:01.697543+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:01.697543+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44627570,"num_objects":5448,"num_object_clones":0,"num_object_copies":21792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5452,"num_write_kb":43596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5532","reported_seq":5593,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.747888+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:01.747888+0000","last_peered":"2024-09-15T03:09:01.747888+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:01.747888+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45274738,"num_objects":5527,"num_object_clones":0,"num_object_copies":22108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":44228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5446","reported_seq":5501,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.895996+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:01.895996+0000","last_peered":"2024-09-15T03:09:01.895996+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:01.895996+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5446,"log_dups_size":0,"ondisk_log_size":5446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44543204,"num_objects":5438,"num_object_clones":0,"num_object_copies":21752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":43521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5451","reported_seq":5534,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:02.420570+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:02.420570+0000","last_peered":"2024-09-15T03:09:02.420570+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:02.420570+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635762,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5449,"num_write_kb":43590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5396","reported_seq":5454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.827041+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:01.827041+0000","last_peered":"2024-09-15T03:09:01.827041+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:01.827041+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44159403,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3313}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5424","reported_seq":8434,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.933590+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:01.933590+0000","last_peered":"2024-09-15T03:09:01.933590+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:01.933590+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44423993,"num_objects":5423,"num_object_clones":0,"num_object_copies":21692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5423,"num_write_kb":43383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1487,"num_bytes_recovered":12165120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":4916}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5526","reported_seq":5603,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:02.505252+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:02.505252+0000","last_peered":"2024-09-15T03:09:02.505252+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:02.505252+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45259577,"num_objects":5525,"num_object_clones":0,"num_object_copies":22100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5525,"num_write_kb":44199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5312","reported_seq":5362,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.707329+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:01.707329+0000","last_peered":"2024-09-15T03:09:01.707329+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:01.707329+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5312,"log_dups_size":0,"ondisk_log_size":5312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43480690,"num_objects":5308,"num_object_clones":0,"num_object_copies":21232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5310,"num_write_kb":42476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5417","reported_seq":5498,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:02.321054+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:02.321054+0000","last_peered":"2024-09-15T03:09:02.321054+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:02.321054+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44349042,"num_objects":5414,"num_object_clones":0,"num_object_copies":21656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":43317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5474","reported_seq":5537,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:01.748457+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:01.748457+0000","last_peered":"2024-09-15T03:09:01.748457+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:01.748457+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5474,"log_dups_size":0,"ondisk_log_size":5474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44799602,"num_objects":5469,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5472,"num_write_kb":43764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3387}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":711391804,"num_objects":86844,"num_object_clones":0,"num_object_copies":347376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86871,"num_write_kb":694850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":12165120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1295585280,"data_stored":1295585280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86898,"ondisk_log_size":86898,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652888,"kb_used_data":346704,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93718952,"statfs":{"total":96636764160,"available":95968206848,"internally_reserved":0,"allocated":355024896,"data_stored":354865918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":5.6669999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":7.8609999999999998},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":4.8220000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":7.9189999999999996},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":4.7919999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":648624,"kb_used_data":347112,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93723216,"statfs":{"total":96636764160,"available":95972573184,"internally_reserved":0,"allocated":355442688,"data_stored":355286856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":5.8860000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.317},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":5.8390000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521128,"kb_used_data":223264,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93850712,"statfs":{"total":96636764160,"available":96103129088,"internally_reserved":0,"allocated":228622336,"data_stored":228463358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.307},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":3.238}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":0.219},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":2.355}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.2730000000000001},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":4.0549999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644112,"kb_used_data":346312,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93727728,"statfs":{"total":96636764160,"available":95977193472,"internally_reserved":0,"allocated":354623488,"data_stored":354464510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":2.556},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":1.972}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":18.808},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":2.2589999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":1.331},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":1.5549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355360768,"data_stored":355360768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228581376,"data_stored":228581376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":356143104,"data_stored":356143104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355500032,"data_stored":355500032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:08.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:08.420 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 212 20 88419 88399 3.25649 1.95312 0.139384 0.0479666 2024-09-15T03:09:09.420 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 213 20 88659 88639 3.25001 1.875 0.0541204 0.048052 2024-09-15T03:09:09.749 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:09.753 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:09.765 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-15T03:09:08.728652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":721730108,"num_objects":88106,"num_object_clones":0,"num_object_copies":352424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88133,"num_write_kb":704946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88160,"ondisk_log_size":88160,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2508212,"kb_used_data":1284052,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":374979148,"statfs":{"total":386547056640,"available":383978647552,"internally_reserved":0,"allocated":1314869248,"data_stored":1314236482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,5,11,15,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31268864,"num_objects":3817,"num_object_clones":0,"num_object_copies":15268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3817,"num_write_kb":30536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005329"},"pg_stats":[{"pgid":"1.f","version":"31'5485","reported_seq":5527,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.782831+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:06.782831+0000","last_peered":"2024-09-15T03:09:06.782831+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:06.782831+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44933120,"num_objects":5485,"num_object_clones":0,"num_object_copies":21940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":43880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5268","reported_seq":5330,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.742797+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:06.742797+0000","last_peered":"2024-09-15T03:09:06.742797+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:06.742797+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5268,"log_dups_size":0,"ondisk_log_size":5268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43120242,"num_objects":5264,"num_object_clones":0,"num_object_copies":21056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5266,"num_write_kb":42124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5639","reported_seq":5720,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:07.445077+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:07.445077+0000","last_peered":"2024-09-15T03:09:07.445077+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:07.445077+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46168889,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5638,"num_write_kb":45094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3534}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5381","reported_seq":5444,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.752472+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:06.752472+0000","last_peered":"2024-09-15T03:09:06.752472+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:06.752472+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5381,"log_dups_size":0,"ondisk_log_size":5381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44045938,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5379,"num_write_kb":43021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5592","reported_seq":5646,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.871067+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:06.871067+0000","last_peered":"2024-09-15T03:09:06.871067+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:06.871067+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45782642,"num_objects":5589,"num_object_clones":0,"num_object_copies":22356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":44717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5547","reported_seq":5591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.900598+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:06.900598+0000","last_peered":"2024-09-15T03:09:06.900598+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:06.900598+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45441024,"num_objects":5547,"num_object_clones":0,"num_object_copies":22188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5547,"num_write_kb":44376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5527","reported_seq":5592,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.526550+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:06.526550+0000","last_peered":"2024-09-15T03:09:06.526550+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:06.526550+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5527,"log_dups_size":0,"ondisk_log_size":5527,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45225586,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5525,"num_write_kb":44180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3469}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5611","reported_seq":5674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.722366+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:06.722366+0000","last_peered":"2024-09-15T03:09:06.722366+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:06.722366+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5611,"log_dups_size":0,"ondisk_log_size":5611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45921906,"num_objects":5606,"num_object_clones":0,"num_object_copies":22424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5609,"num_write_kb":44860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5528","reported_seq":5585,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.756042+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:06.756042+0000","last_peered":"2024-09-15T03:09:06.756042+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:06.756042+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5528,"log_dups_size":0,"ondisk_log_size":5528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45214948,"num_objects":5520,"num_object_clones":0,"num_object_copies":22080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5525,"num_write_kb":44177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3478}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5532","reported_seq":5615,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:07.542052+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:07.542052+0000","last_peered":"2024-09-15T03:09:07.542052+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:07.542052+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45299314,"num_objects":5530,"num_object_clones":0,"num_object_copies":22120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":44238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3472}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5472","reported_seq":5530,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.858619+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:06.858619+0000","last_peered":"2024-09-15T03:09:06.858619+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:06.858619+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44781995,"num_objects":5467,"num_object_clones":0,"num_object_copies":21868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5469,"num_write_kb":43740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5516","reported_seq":8604,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.919222+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:06.919222+0000","last_peered":"2024-09-15T03:09:06.919222+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:06.919222+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5516,"log_dups_size":0,"ondisk_log_size":5516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45177657,"num_objects":5515,"num_object_clones":0,"num_object_copies":22060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5515,"num_write_kb":44119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1526,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5047}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5609","reported_seq":5686,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:07.486853+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:07.486853+0000","last_peered":"2024-09-15T03:09:07.486853+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:07.486853+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5609,"log_dups_size":0,"ondisk_log_size":5609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45939513,"num_objects":5608,"num_object_clones":0,"num_object_copies":22432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":44863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3569}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5398","reported_seq":5448,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.875845+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:06.875845+0000","last_peered":"2024-09-15T03:09:06.875845+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:06.875845+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5398,"log_dups_size":0,"ondisk_log_size":5398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44185202,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5396,"num_write_kb":43164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5502","reported_seq":5583,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:07.466780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:07.466780+0000","last_peered":"2024-09-15T03:09:07.466780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:07.466780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5502,"log_dups_size":0,"ondisk_log_size":5502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45045362,"num_objects":5499,"num_object_clones":0,"num_object_copies":21996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5500,"num_write_kb":43997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3511}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5553","reported_seq":5616,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:06.699701+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:06.699701+0000","last_peered":"2024-09-15T03:09:06.699701+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:06.699701+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45446770,"num_objects":5548,"num_object_clones":0,"num_object_copies":22192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":44396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":721730108,"num_objects":88106,"num_object_clones":0,"num_object_copies":352424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88133,"num_write_kb":704946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1552,"num_bytes_recovered":12484608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1316446208,"data_stored":1316446208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88160,"ondisk_log_size":88160,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":666380,"kb_used_data":351876,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93705460,"statfs":{"total":96636764160,"available":95954391040,"internally_reserved":0,"allocated":360321024,"data_stored":360162046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.29599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.653,"15min":1.639},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":13.176,"5min":13.176,"15min":13.176},"last":1.4970000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28199999999999997,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":5.7699999999999996,"5min":5.7699999999999996,"15min":5.7699999999999996},"last":0.20300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":1.528},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.198}]}]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":657812,"kb_used_data":352140,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93714028,"statfs":{"total":96636764160,"available":95963164672,"internally_reserved":0,"allocated":360591360,"data_stored":360435528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.319}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":0.29099999999999998},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":0.32800000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.34599999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530592,"kb_used_data":228568,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93841248,"statfs":{"total":96636764160,"available":96093437952,"internally_reserved":0,"allocated":234053632,"data_stored":233894654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.9570000000000001,"15min":2.1629999999999998},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":4.0019999999999998,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.193},{"interface":"front","average":{"1min":1.845,"5min":1.526,"15min":1.4730000000000001},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":6.7939999999999996,"5min":9.4339999999999993,"15min":9.4339999999999993},"last":0.20799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.2909999999999999,"15min":1.367},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.5910000000000002,"5min":9.1620000000000008,"15min":9.1620000000000008},"last":3.347},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.5309999999999999,"15min":1.671},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.7160000000000002,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":1.4079999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.54,"15min":1.619},"min":{"1min":0.246,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.7160000000000002,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.251},{"interface":"front","average":{"1min":1.399,"5min":1.258,"15min":1.234},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":6.8959999999999999,"5min":6.8959999999999999,"15min":6.8959999999999999},"last":3.2080000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653428,"kb_used_data":351468,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93718412,"statfs":{"total":96636764160,"available":95967653888,"internally_reserved":0,"allocated":359903232,"data_stored":359744254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.17499999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.70699999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":1.9790000000000001},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":2.1219999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":360529920,"data_stored":360529920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233988096,"data_stored":233988096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361259008,"data_stored":361259008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":360669184,"data_stored":360669184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:09.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:09:09.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5485", 'reported_seq': 5527, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.782831+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:09:06.782831+0000', 'last_peered': '2024-09-15T03:09:06.782831+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:09:06.782831+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5485, 'log_dups_size': 0, 'ondisk_log_size': 5485, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44933120, 'num_objects': 5485, 'num_object_clones': 0, 'num_object_copies': 21940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5485, 'num_write_kb': 43880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3509}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:09.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:09:09.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5268", 'reported_seq': 5330, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.742797+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:09:06.742797+0000', 'last_peered': '2024-09-15T03:09:06.742797+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:09:06.742797+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5268, 'log_dups_size': 0, 'ondisk_log_size': 5268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43120242, 'num_objects': 5264, 'num_object_clones': 0, 'num_object_copies': 21056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5264, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5266, 'num_write_kb': 42124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3266}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5639", 'reported_seq': 5720, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:07.445077+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:09:07.445077+0000', 'last_peered': '2024-09-15T03:09:07.445077+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:09:07.445077+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5639, 'log_dups_size': 0, 'ondisk_log_size': 5639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46168889, 'num_objects': 5636, 'num_object_clones': 0, 'num_object_copies': 22544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5638, 'num_write_kb': 45094, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3534}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5381", 'reported_seq': 5444, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.752472+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:09:06.752472+0000', 'last_peered': '2024-09-15T03:09:06.752472+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:09:06.752472+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5381, 'log_dups_size': 0, 'ondisk_log_size': 5381, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44045938, 'num_objects': 5377, 'num_object_clones': 0, 'num_object_copies': 21508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5379, 'num_write_kb': 43021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5592", 'reported_seq': 5646, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.871067+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:09:06.871067+0000', 'last_peered': '2024-09-15T03:09:06.871067+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:09:06.871067+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5592, 'log_dups_size': 0, 'ondisk_log_size': 5592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45782642, 'num_objects': 5589, 'num_object_clones': 0, 'num_object_copies': 22356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5590, 'num_write_kb': 44717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3482}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5547", 'reported_seq': 5591, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.900598+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:09:06.900598+0000', 'last_peered': '2024-09-15T03:09:06.900598+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:09:06.900598+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5547, 'log_dups_size': 0, 'ondisk_log_size': 5547, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45441024, 'num_objects': 5547, 'num_object_clones': 0, 'num_object_copies': 22188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5547, 'num_write_kb': 44376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3491}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:09:09.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5527", 'reported_seq': 5592, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.526550+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:09:06.526550+0000', 'last_peered': '2024-09-15T03:09:06.526550+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:09:06.526550+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5527, 'log_dups_size': 0, 'ondisk_log_size': 5527, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45225586, 'num_objects': 5521, 'num_object_clones': 0, 'num_object_copies': 22084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5525, 'num_write_kb': 44180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3469}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5611", 'reported_seq': 5674, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.722366+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:09:06.722366+0000', 'last_peered': '2024-09-15T03:09:06.722366+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:09:06.722366+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5611, 'log_dups_size': 0, 'ondisk_log_size': 5611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45921906, 'num_objects': 5606, 'num_object_clones': 0, 'num_object_copies': 22424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5609, 'num_write_kb': 44860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5528", 'reported_seq': 5585, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.756042+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:09:06.756042+0000', 'last_peered': '2024-09-15T03:09:06.756042+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:09:06.756042+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5528, 'log_dups_size': 0, 'ondisk_log_size': 5528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45214948, 'num_objects': 5520, 'num_object_clones': 0, 'num_object_copies': 22080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5525, 'num_write_kb': 44177, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3478}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5532", 'reported_seq': 5615, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:07.542052+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:09:07.542052+0000', 'last_peered': '2024-09-15T03:09:07.542052+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:09:07.542052+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5532, 'log_dups_size': 0, 'ondisk_log_size': 5532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45299314, 'num_objects': 5530, 'num_object_clones': 0, 'num_object_copies': 22120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5530, 'num_write_kb': 44238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3472}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5472", 'reported_seq': 5530, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.858619+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:09:06.858619+0000', 'last_peered': '2024-09-15T03:09:06.858619+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:09:06.858619+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5472, 'log_dups_size': 0, 'ondisk_log_size': 5472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44781995, 'num_objects': 5467, 'num_object_clones': 0, 'num_object_copies': 21868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5469, 'num_write_kb': 43740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3389}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:09.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5516", 'reported_seq': 8604, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.919222+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:09:06.919222+0000', 'last_peered': '2024-09-15T03:09:06.919222+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:09:06.919222+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5516, 'log_dups_size': 0, 'ondisk_log_size': 5516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45177657, 'num_objects': 5515, 'num_object_clones': 0, 'num_object_copies': 22060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5515, 'num_write_kb': 44119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1526, 'num_bytes_recovered': 12484608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5047}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5609", 'reported_seq': 5686, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:07.486853+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:09:07.486853+0000', 'last_peered': '2024-09-15T03:09:07.486853+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:09:07.486853+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5609, 'log_dups_size': 0, 'ondisk_log_size': 5609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45939513, 'num_objects': 5608, 'num_object_clones': 0, 'num_object_copies': 22432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5608, 'num_write_kb': 44863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5398", 'reported_seq': 5448, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.875845+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:09:06.875845+0000', 'last_peered': '2024-09-15T03:09:06.875845+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:09:06.875845+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5398, 'log_dups_size': 0, 'ondisk_log_size': 5398, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44185202, 'num_objects': 5394, 'num_object_clones': 0, 'num_object_copies': 21576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5396, 'num_write_kb': 43164, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5502", 'reported_seq': 5583, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:07.466780+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:09:07.466780+0000', 'last_peered': '2024-09-15T03:09:07.466780+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:09:07.466780+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5502, 'log_dups_size': 0, 'ondisk_log_size': 5502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45045362, 'num_objects': 5499, 'num_object_clones': 0, 'num_object_copies': 21996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5500, 'num_write_kb': 43997, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:09:09.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5553", 'reported_seq': 5616, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:06.699701+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:09:06.699701+0000', 'last_peered': '2024-09-15T03:09:06.699701+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:09:06.699701+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5553, 'log_dups_size': 0, 'ondisk_log_size': 5553, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45446770, 'num_objects': 5548, 'num_object_clones': 0, 'num_object_copies': 22192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5551, 'num_write_kb': 44396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:09.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:09:10.426 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 214 20 88906 88886 3.24376 1.92969 0.101932 0.0481539 2024-09-15T03:09:11.431 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 215 20 89155 89135 3.23772 1.94531 0.0370142 0.0482413 2024-09-15T03:09:11.543 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:11.543 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31132/352424 objects degraded (8.834%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":274,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88106,"data_bytes":721730108,"bytes_used":2568409088,"bytes_avail":383978647552,"bytes_total":386547056640,"degraded_objects":31132,"degraded_total":352424,"degraded_ratio":0.088336776155993918,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":64824,"recovering_keys_per_sec":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"write_bytes_sec":1729095,"read_op_per_sec":0,"write_op_per_sec":211},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:09:11.599 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:09:11.599 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:12.426 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 216 20 89401 89381 3.23163 1.92188 0.0565178 0.0483262 2024-09-15T03:09:12.819 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:12.837 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:12.842 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-15T03:09:12.735330+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":731871804,"num_objects":89344,"num_object_clones":0,"num_object_copies":357376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89371,"num_write_kb":714850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1612,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89398,"ondisk_log_size":89398,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2538036,"kb_used_data":1305556,"kb_used_omap":14,"kb_used_meta":1232433,"kb_avail":374949324,"statfs":{"total":386547056640,"available":383948107776,"internally_reserved":0,"allocated":1336889344,"data_stored":1336256578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1262011419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,9,6,12,14,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30900224,"num_objects":3772,"num_object_clones":0,"num_object_copies":15088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3772,"num_write_kb":30176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1269760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011005"},"pg_stats":[{"pgid":"1.f","version":"31'5570","reported_seq":5612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.941737+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:11.941737+0000","last_peered":"2024-09-15T03:09:11.941737+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:11.941737+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45629440,"num_objects":5570,"num_object_clones":0,"num_object_copies":22280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5570,"num_write_kb":44560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5340","reported_seq":5402,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.649984+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:11.649984+0000","last_peered":"2024-09-15T03:09:11.649984+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:11.649984+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43710066,"num_objects":5336,"num_object_clones":0,"num_object_copies":21344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5732","reported_seq":5813,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.439089+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:12.439089+0000","last_peered":"2024-09-15T03:09:12.439089+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:12.439089+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46930745,"num_objects":5729,"num_object_clones":0,"num_object_copies":22916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":45838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3627}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5455","reported_seq":5518,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.714206+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:11.714206+0000","last_peered":"2024-09-15T03:09:11.714206+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:11.714206+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44652146,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":43613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5671","reported_seq":5727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.802735+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:11.802735+0000","last_peered":"2024-09-15T03:09:11.802735+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:11.802735+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46429810,"num_objects":5668,"num_object_clones":0,"num_object_copies":22672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":45349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5621","reported_seq":5664,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.992796+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:11.992796+0000","last_peered":"2024-09-15T03:09:11.992796+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:11.992796+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5621,"log_dups_size":0,"ondisk_log_size":5621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46047232,"num_objects":5621,"num_object_clones":0,"num_object_copies":22484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5621,"num_write_kb":44968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5606","reported_seq":5671,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.759465+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:11.759465+0000","last_peered":"2024-09-15T03:09:11.759465+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:11.759465+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45872754,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5604,"num_write_kb":44812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3548}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5672","reported_seq":5735,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.699318+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:11.699318+0000","last_peered":"2024-09-15T03:09:11.699318+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:11.699318+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421618,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5670,"num_write_kb":45348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5608","reported_seq":5665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.976362+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:11.976362+0000","last_peered":"2024-09-15T03:09:11.976362+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:11.976362+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45870308,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5617","reported_seq":5700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.583093+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:12.583093+0000","last_peered":"2024-09-15T03:09:12.583093+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:12.583093+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45995634,"num_objects":5615,"num_object_clones":0,"num_object_copies":22460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":44918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3557}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5562","reported_seq":5620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.938241+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:11.938241+0000","last_peered":"2024-09-15T03:09:11.938241+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:11.938241+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45519275,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5589","reported_seq":8797,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.958135+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:11.958135+0000","last_peered":"2024-09-15T03:09:11.958135+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:11.958135+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45775673,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5588,"num_write_kb":44703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1586,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5180}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5684","reported_seq":5758,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.553598+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:12.553598+0000","last_peered":"2024-09-15T03:09:12.553598+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:12.553598+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5684,"log_dups_size":0,"ondisk_log_size":5684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46553913,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":45463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5476","reported_seq":5525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.025670+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:12.025670+0000","last_peered":"2024-09-15T03:09:12.025670+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:12.025670+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44824178,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5581","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.387241+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:12.387241+0000","last_peered":"2024-09-15T03:09:12.387241+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:12.387241+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45692530,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5614","reported_seq":5676,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.717269+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:11.717269+0000","last_peered":"2024-09-15T03:09:11.717269+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:11.717269+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5614,"log_dups_size":0,"ondisk_log_size":5614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45946482,"num_objects":5609,"num_object_clones":0,"num_object_copies":22436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5612,"num_write_kb":44884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":731871804,"num_objects":89344,"num_object_clones":0,"num_object_copies":357376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89371,"num_write_kb":714850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1612,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337114624,"data_stored":1337114624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89398,"ondisk_log_size":89398,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":658764,"kb_used_data":356740,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93713076,"statfs":{"total":96636764160,"available":95962189824,"internally_reserved":0,"allocated":365301760,"data_stored":365142782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1,2,8,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.26400000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.23899999999999999},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.93799999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667800,"kb_used_data":357968,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93704040,"statfs":{"total":96636764160,"available":95952936960,"internally_reserved":0,"allocated":366559232,"data_stored":366403400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,4,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":16.687999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":4.415}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":1.3129999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":1.349}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":8.0060000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":4.4509999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544040,"kb_used_data":233696,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93827800,"statfs":{"total":96636764160,"available":96079667200,"internally_reserved":0,"allocated":239304704,"data_stored":239145726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.317}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.376},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.499}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.222},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.33700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667432,"kb_used_data":357152,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93704408,"statfs":{"total":96636764160,"available":95953313792,"internally_reserved":0,"allocated":365723648,"data_stored":365564670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.625},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":3.488},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":365576192,"data_stored":365576192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239362048,"data_stored":239362048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366350336,"data_stored":366350336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":365826048,"data_stored":365826048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:13.426 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 217 20 89658 89638 3.22599 2.00781 0.0272671 0.0484199 2024-09-15T03:09:14.426 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 218 20 89920 89900 3.22058 2.04688 0.0603273 0.0485018 2024-09-15T03:09:15.427 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 219 20 90179 90159 3.2151 2.02344 0.0625388 0.0485822 2024-09-15T03:09:15.844 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:16.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:2024-09-15T03:09:16.431797+0000 min lat: 0.00240791 max lat: 2.51529 avg lat: 0.0486687 2024-09-15T03:09:16.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T03:09:16.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 220 20 90429 90409 3.20929 1.95312 0.0928243 0.0486687 2024-09-15T03:09:16.997 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:17.015 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:17.054 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-15T03:09:16.736325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":731871804,"num_objects":89344,"num_object_clones":0,"num_object_copies":357376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89371,"num_write_kb":714850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1612,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89398,"ondisk_log_size":89398,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2538036,"kb_used_data":1305556,"kb_used_omap":14,"kb_used_meta":1232433,"kb_avail":374949324,"statfs":{"total":386547056640,"available":383948107776,"internally_reserved":0,"allocated":1336889344,"data_stored":1336256578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1262011419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,9,6,12,14,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20480000,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":20000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008647"},"pg_stats":[{"pgid":"1.f","version":"31'5570","reported_seq":5612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.941737+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:11.941737+0000","last_peered":"2024-09-15T03:09:11.941737+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:11.941737+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45629440,"num_objects":5570,"num_object_clones":0,"num_object_copies":22280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5570,"num_write_kb":44560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5340","reported_seq":5402,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.649984+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:11.649984+0000","last_peered":"2024-09-15T03:09:11.649984+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:11.649984+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43710066,"num_objects":5336,"num_object_clones":0,"num_object_copies":21344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5732","reported_seq":5813,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.439089+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:12.439089+0000","last_peered":"2024-09-15T03:09:12.439089+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:12.439089+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46930745,"num_objects":5729,"num_object_clones":0,"num_object_copies":22916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":45838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3627}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5455","reported_seq":5518,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.714206+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:11.714206+0000","last_peered":"2024-09-15T03:09:11.714206+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:11.714206+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44652146,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":43613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5671","reported_seq":5727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.802735+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:11.802735+0000","last_peered":"2024-09-15T03:09:11.802735+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:11.802735+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46429810,"num_objects":5668,"num_object_clones":0,"num_object_copies":22672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":45349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5621","reported_seq":5664,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.992796+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:11.992796+0000","last_peered":"2024-09-15T03:09:11.992796+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:11.992796+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5621,"log_dups_size":0,"ondisk_log_size":5621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46047232,"num_objects":5621,"num_object_clones":0,"num_object_copies":22484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5621,"num_write_kb":44968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5606","reported_seq":5671,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.759465+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:11.759465+0000","last_peered":"2024-09-15T03:09:11.759465+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:11.759465+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45872754,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5604,"num_write_kb":44812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3548}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5672","reported_seq":5735,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.699318+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:11.699318+0000","last_peered":"2024-09-15T03:09:11.699318+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:11.699318+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421618,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5670,"num_write_kb":45348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5608","reported_seq":5665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.976362+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:11.976362+0000","last_peered":"2024-09-15T03:09:11.976362+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:11.976362+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45870308,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5617","reported_seq":5700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.583093+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:12.583093+0000","last_peered":"2024-09-15T03:09:12.583093+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:12.583093+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45995634,"num_objects":5615,"num_object_clones":0,"num_object_copies":22460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":44918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3557}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5562","reported_seq":5620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.938241+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:11.938241+0000","last_peered":"2024-09-15T03:09:11.938241+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:11.938241+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45519275,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5589","reported_seq":8797,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.958135+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:11.958135+0000","last_peered":"2024-09-15T03:09:11.958135+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:11.958135+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45775673,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5588,"num_write_kb":44703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1586,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5180}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5684","reported_seq":5758,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.553598+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:12.553598+0000","last_peered":"2024-09-15T03:09:12.553598+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:12.553598+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5684,"log_dups_size":0,"ondisk_log_size":5684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46553913,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":45463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5476","reported_seq":5525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.025670+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:12.025670+0000","last_peered":"2024-09-15T03:09:12.025670+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:12.025670+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44824178,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5581","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.387241+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:12.387241+0000","last_peered":"2024-09-15T03:09:12.387241+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:12.387241+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45692530,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5614","reported_seq":5676,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.717269+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:11.717269+0000","last_peered":"2024-09-15T03:09:11.717269+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:11.717269+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5614,"log_dups_size":0,"ondisk_log_size":5614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45946482,"num_objects":5609,"num_object_clones":0,"num_object_copies":22436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5612,"num_write_kb":44884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":731871804,"num_objects":89344,"num_object_clones":0,"num_object_copies":357376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89371,"num_write_kb":714850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1612,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337114624,"data_stored":1337114624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89398,"ondisk_log_size":89398,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":658764,"kb_used_data":356740,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93713076,"statfs":{"total":96636764160,"available":95962189824,"internally_reserved":0,"allocated":365301760,"data_stored":365142782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1,2,8,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.26400000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.23899999999999999},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.93799999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667800,"kb_used_data":357968,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93704040,"statfs":{"total":96636764160,"available":95952936960,"internally_reserved":0,"allocated":366559232,"data_stored":366403400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,4,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":16.687999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":4.415}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":1.3129999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":1.349}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":8.0060000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":4.4509999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544040,"kb_used_data":233696,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93827800,"statfs":{"total":96636764160,"available":96079667200,"internally_reserved":0,"allocated":239304704,"data_stored":239145726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.317}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.376},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.499}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.222},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.33700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667432,"kb_used_data":357152,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93704408,"statfs":{"total":96636764160,"available":95953313792,"internally_reserved":0,"allocated":365723648,"data_stored":365564670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.625},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":3.488},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":365576192,"data_stored":365576192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239362048,"data_stored":239362048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366350336,"data_stored":366350336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":365826048,"data_stored":365826048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:17.056 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:17.436 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 221 20 90682 90662 3.20372 1.97656 0.0476782 0.0487573 2024-09-15T03:09:18.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 222 19 90923 90904 3.1978 1.89062 0.0788408 0.0488494 2024-09-15T03:09:18.717 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:18.742 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:18.766 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-15T03:09:16.736325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":731871804,"num_objects":89344,"num_object_clones":0,"num_object_copies":357376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89371,"num_write_kb":714850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1612,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89398,"ondisk_log_size":89398,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2538036,"kb_used_data":1305556,"kb_used_omap":14,"kb_used_meta":1232433,"kb_avail":374949324,"statfs":{"total":386547056640,"available":383948107776,"internally_reserved":0,"allocated":1336889344,"data_stored":1336256578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1262011419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,9,6,12,14,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20480000,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":20000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008647"},"pg_stats":[{"pgid":"1.f","version":"31'5570","reported_seq":5612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.941737+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:11.941737+0000","last_peered":"2024-09-15T03:09:11.941737+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:11.941737+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45629440,"num_objects":5570,"num_object_clones":0,"num_object_copies":22280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5570,"num_write_kb":44560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5340","reported_seq":5402,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.649984+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:11.649984+0000","last_peered":"2024-09-15T03:09:11.649984+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:11.649984+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43710066,"num_objects":5336,"num_object_clones":0,"num_object_copies":21344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5732","reported_seq":5813,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.439089+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:12.439089+0000","last_peered":"2024-09-15T03:09:12.439089+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:12.439089+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5732,"log_dups_size":0,"ondisk_log_size":5732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46930745,"num_objects":5729,"num_object_clones":0,"num_object_copies":22916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":45838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3627}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5455","reported_seq":5518,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.714206+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:11.714206+0000","last_peered":"2024-09-15T03:09:11.714206+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:11.714206+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44652146,"num_objects":5451,"num_object_clones":0,"num_object_copies":21804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5453,"num_write_kb":43613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3508}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5671","reported_seq":5727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.802735+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:11.802735+0000","last_peered":"2024-09-15T03:09:11.802735+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:11.802735+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46429810,"num_objects":5668,"num_object_clones":0,"num_object_copies":22672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":45349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5621","reported_seq":5664,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.992796+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:11.992796+0000","last_peered":"2024-09-15T03:09:11.992796+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:11.992796+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5621,"log_dups_size":0,"ondisk_log_size":5621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46047232,"num_objects":5621,"num_object_clones":0,"num_object_copies":22484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5621,"num_write_kb":44968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3565}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5606","reported_seq":5671,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.759465+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:11.759465+0000","last_peered":"2024-09-15T03:09:11.759465+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:11.759465+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45872754,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5604,"num_write_kb":44812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3548}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5672","reported_seq":5735,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.699318+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:11.699318+0000","last_peered":"2024-09-15T03:09:11.699318+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:11.699318+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5672,"log_dups_size":0,"ondisk_log_size":5672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421618,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5670,"num_write_kb":45348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3586}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5608","reported_seq":5665,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.976362+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:11.976362+0000","last_peered":"2024-09-15T03:09:11.976362+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:11.976362+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45870308,"num_objects":5600,"num_object_clones":0,"num_object_copies":22400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5617","reported_seq":5700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.583093+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:12.583093+0000","last_peered":"2024-09-15T03:09:12.583093+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:12.583093+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5617,"log_dups_size":0,"ondisk_log_size":5617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45995634,"num_objects":5615,"num_object_clones":0,"num_object_copies":22460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":44918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3557}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5562","reported_seq":5620,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.938241+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:11.938241+0000","last_peered":"2024-09-15T03:09:11.938241+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:11.938241+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45519275,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5559,"num_write_kb":44460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5589","reported_seq":8797,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.958135+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:11.958135+0000","last_peered":"2024-09-15T03:09:11.958135+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:11.958135+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45775673,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5588,"num_write_kb":44703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1586,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5180}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5684","reported_seq":5758,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.553598+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:12.553598+0000","last_peered":"2024-09-15T03:09:12.553598+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:12.553598+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5684,"log_dups_size":0,"ondisk_log_size":5684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46553913,"num_objects":5683,"num_object_clones":0,"num_object_copies":22732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5683,"num_write_kb":45463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5476","reported_seq":5525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.025670+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:12.025670+0000","last_peered":"2024-09-15T03:09:12.025670+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:12.025670+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44824178,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5474,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5581","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:12.387241+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:12.387241+0000","last_peered":"2024-09-15T03:09:12.387241+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:12.387241+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45692530,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5614","reported_seq":5676,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:11.717269+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:11.717269+0000","last_peered":"2024-09-15T03:09:11.717269+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:11.717269+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5614,"log_dups_size":0,"ondisk_log_size":5614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45946482,"num_objects":5609,"num_object_clones":0,"num_object_copies":22436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5612,"num_write_kb":44884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":731871804,"num_objects":89344,"num_object_clones":0,"num_object_copies":357376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89371,"num_write_kb":714850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1612,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337114624,"data_stored":1337114624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89398,"ondisk_log_size":89398,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":658764,"kb_used_data":356740,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93713076,"statfs":{"total":96636764160,"available":95962189824,"internally_reserved":0,"allocated":365301760,"data_stored":365142782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,1,2,8,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.26400000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.23899999999999999},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.93799999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667800,"kb_used_data":357968,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93704040,"statfs":{"total":96636764160,"available":95952936960,"internally_reserved":0,"allocated":366559232,"data_stored":366403400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,4,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":16.687999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":4.415}]},{"osd":1,"last update":"Sun Sep 15 03:08:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8009999999999999,"5min":1.2030000000000001,"15min":1.103},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":19.343,"5min":19.343,"15min":19.343},"last":1.3129999999999999},{"interface":"front","average":{"1min":2.7509999999999999,"5min":1.421,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":23.311,"5min":23.311,"15min":23.311},"last":1.349}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":8.0060000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":4.4509999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":544040,"kb_used_data":233696,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93827800,"statfs":{"total":96636764160,"available":96079667200,"internally_reserved":0,"allocated":239304704,"data_stored":239145726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.317}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.376},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.499}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.222},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.33700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667432,"kb_used_data":357152,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93704408,"statfs":{"total":96636764160,"available":95953313792,"internally_reserved":0,"allocated":365723648,"data_stored":365564670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:08:09 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.0229999999999999,"15min":0.84299999999999997},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.527000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.625},{"interface":"front","average":{"1min":1.498,"5min":1.5149999999999999,"15min":1.518},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.247},"max":{"1min":10.675000000000001,"5min":10.675000000000001,"15min":10.675000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":3.488},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":365576192,"data_stored":365576192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":239362048,"data_stored":239362048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366350336,"data_stored":366350336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":365826048,"data_stored":365826048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:18.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:09:18.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5570", 'reported_seq': 5612, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.941737+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:09:11.941737+0000', 'last_peered': '2024-09-15T03:09:11.941737+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:09:11.941737+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5570, 'log_dups_size': 0, 'ondisk_log_size': 5570, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45629440, 'num_objects': 5570, 'num_object_clones': 0, 'num_object_copies': 22280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5570, 'num_write_kb': 44560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3594}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:18.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5340", 'reported_seq': 5402, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.649984+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:09:11.649984+0000', 'last_peered': '2024-09-15T03:09:11.649984+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:09:11.649984+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5340, 'log_dups_size': 0, 'ondisk_log_size': 5340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43710066, 'num_objects': 5336, 'num_object_clones': 0, 'num_object_copies': 21344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5336, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5338, 'num_write_kb': 42700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3338}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5732", 'reported_seq': 5813, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:12.439089+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:09:12.439089+0000', 'last_peered': '2024-09-15T03:09:12.439089+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:09:12.439089+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5732, 'log_dups_size': 0, 'ondisk_log_size': 5732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46930745, 'num_objects': 5729, 'num_object_clones': 0, 'num_object_copies': 22916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5731, 'num_write_kb': 45838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3627}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5455", 'reported_seq': 5518, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.714206+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:09:11.714206+0000', 'last_peered': '2024-09-15T03:09:11.714206+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:09:11.714206+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5455, 'log_dups_size': 0, 'ondisk_log_size': 5455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44652146, 'num_objects': 5451, 'num_object_clones': 0, 'num_object_copies': 21804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5453, 'num_write_kb': 43613, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3508}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5671", 'reported_seq': 5727, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.802735+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:09:11.802735+0000', 'last_peered': '2024-09-15T03:09:11.802735+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:09:11.802735+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5671, 'log_dups_size': 0, 'ondisk_log_size': 5671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46429810, 'num_objects': 5668, 'num_object_clones': 0, 'num_object_copies': 22672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5669, 'num_write_kb': 45349, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3561}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5621", 'reported_seq': 5664, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.992796+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:09:11.992796+0000', 'last_peered': '2024-09-15T03:09:11.992796+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:09:11.992796+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5621, 'log_dups_size': 0, 'ondisk_log_size': 5621, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46047232, 'num_objects': 5621, 'num_object_clones': 0, 'num_object_copies': 22484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5621, 'num_write_kb': 44968, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3565}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:18.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5606", 'reported_seq': 5671, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.759465+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:09:11.759465+0000', 'last_peered': '2024-09-15T03:09:11.759465+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:09:11.759465+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5606, 'log_dups_size': 0, 'ondisk_log_size': 5606, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45872754, 'num_objects': 5600, 'num_object_clones': 0, 'num_object_copies': 22400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5604, 'num_write_kb': 44812, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5672", 'reported_seq': 5735, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.699318+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:09:11.699318+0000', 'last_peered': '2024-09-15T03:09:11.699318+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:09:11.699318+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5672, 'log_dups_size': 0, 'ondisk_log_size': 5672, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46421618, 'num_objects': 5667, 'num_object_clones': 0, 'num_object_copies': 22668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5670, 'num_write_kb': 45348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5608", 'reported_seq': 5665, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.976362+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:09:11.976362+0000', 'last_peered': '2024-09-15T03:09:11.976362+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:09:11.976362+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5608, 'log_dups_size': 0, 'ondisk_log_size': 5608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45870308, 'num_objects': 5600, 'num_object_clones': 0, 'num_object_copies': 22400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5605, 'num_write_kb': 44817, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3558}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5617", 'reported_seq': 5700, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:12.583093+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:09:12.583093+0000', 'last_peered': '2024-09-15T03:09:12.583093+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:09:12.583093+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5617, 'log_dups_size': 0, 'ondisk_log_size': 5617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45995634, 'num_objects': 5615, 'num_object_clones': 0, 'num_object_copies': 22460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5615, 'num_write_kb': 44918, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3557}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5562", 'reported_seq': 5620, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.938241+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:09:11.938241+0000', 'last_peered': '2024-09-15T03:09:11.938241+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:09:11.938241+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5562, 'log_dups_size': 0, 'ondisk_log_size': 5562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45519275, 'num_objects': 5557, 'num_object_clones': 0, 'num_object_copies': 22228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5559, 'num_write_kb': 44460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3479}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:09:18.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5589", 'reported_seq': 8797, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.958135+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:09:11.958135+0000', 'last_peered': '2024-09-15T03:09:11.958135+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:09:11.958135+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5589, 'log_dups_size': 0, 'ondisk_log_size': 5589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45775673, 'num_objects': 5588, 'num_object_clones': 0, 'num_object_copies': 22352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 408, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5588, 'num_write_kb': 44703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1586, 'num_bytes_recovered': 12976128, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:18.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:09:18.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5684", 'reported_seq': 5758, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:12.553598+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:09:12.553598+0000', 'last_peered': '2024-09-15T03:09:12.553598+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:09:12.553598+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5684, 'log_dups_size': 0, 'ondisk_log_size': 5684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46553913, 'num_objects': 5683, 'num_object_clones': 0, 'num_object_copies': 22732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5683, 'num_write_kb': 45463, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3644}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:18.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:09:18.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5476", 'reported_seq': 5525, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:12.025670+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:09:12.025670+0000', 'last_peered': '2024-09-15T03:09:12.025670+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:09:12.025670+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5476, 'log_dups_size': 0, 'ondisk_log_size': 5476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44824178, 'num_objects': 5472, 'num_object_clones': 0, 'num_object_copies': 21888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5474, 'num_write_kb': 43788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3436}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:18.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:09:18.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5581", 'reported_seq': 5662, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:12.387241+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:09:12.387241+0000', 'last_peered': '2024-09-15T03:09:12.387241+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:09:12.387241+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5581, 'log_dups_size': 0, 'ondisk_log_size': 5581, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45692530, 'num_objects': 5578, 'num_object_clones': 0, 'num_object_copies': 22312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5579, 'num_write_kb': 44629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:18.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:09:18.770 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5614", 'reported_seq': 5676, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:11.717269+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:09:11.717269+0000', 'last_peered': '2024-09-15T03:09:11.717269+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:09:11.717269+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5614, 'log_dups_size': 0, 'ondisk_log_size': 5614, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45946482, 'num_objects': 5609, 'num_object_clones': 0, 'num_object_copies': 22436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5612, 'num_write_kb': 44884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:18.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:09:19.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 223 20 91168 91148 3.19201 1.90625 0.148697 0.0489352 2024-09-15T03:09:20.127 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:20.127 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31017/362488 objects degraded (8.557%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":283,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":90622,"data_bytes":742341180,"bytes_used":2585075712,"bytes_avail":383961980928,"bytes_total":386547056640,"degraded_objects":31017,"degraded_total":362488,"degraded_ratio":0.085566970492816308,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105055,"recovering_keys_per_sec":0,"num_objects_recovered":154,"num_bytes_recovered":1261568,"num_keys_recovered":0,"write_bytes_sec":2577262,"read_op_per_sec":0,"write_op_per_sec":314},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:09:20.205 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:09:20.205 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:20.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 224 20 91411 91391 3.18624 1.89844 0.0132167 0.0490257 2024-09-15T03:09:21.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 225 20 91650 91630 3.18038 1.86719 0.0724744 0.0491127 2024-09-15T03:09:21.456 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:21.476 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:21.527 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-15T03:09:20.737307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":742341180,"num_objects":90622,"num_object_clones":0,"num_object_copies":362488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90649,"num_write_kb":725074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":13426688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90676,"ondisk_log_size":90676,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2524488,"kb_used_data":1325288,"kb_used_omap":14,"kb_used_meta":1199153,"kb_avail":374962872,"statfs":{"total":386547056640,"available":383961980928,"internally_reserved":0,"allocated":1357094912,"data_stored":1356462146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227932699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,4,5,14,17,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20611072,"num_objects":2516,"num_object_clones":0,"num_object_copies":10064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":20128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008654"},"pg_stats":[{"pgid":"1.f","version":"31'5654","reported_seq":5695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.043157+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:17.043157+0000","last_peered":"2024-09-15T03:09:17.043157+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:17.043157+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46317568,"num_objects":5654,"num_object_clones":0,"num_object_copies":22616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5654,"num_write_kb":45232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5430","reported_seq":5492,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:16.704066+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:16.704066+0000","last_peered":"2024-09-15T03:09:16.704066+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:16.704066+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5430,"log_dups_size":0,"ondisk_log_size":5430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44447346,"num_objects":5426,"num_object_clones":0,"num_object_copies":21704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5428,"num_write_kb":43420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3428}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5812","reported_seq":5893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.350180+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:17.350180+0000","last_peered":"2024-09-15T03:09:17.350180+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:17.350180+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5812,"log_dups_size":0,"ondisk_log_size":5812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47586105,"num_objects":5809,"num_object_clones":0,"num_object_copies":23236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":46478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3707}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5520","reported_seq":5583,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:16.384974+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:16.384974+0000","last_peered":"2024-09-15T03:09:16.384974+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:16.384974+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45184626,"num_objects":5516,"num_object_clones":0,"num_object_copies":22064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5518,"num_write_kb":44133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3573}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5754","reported_seq":5810,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.009761+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:17.009761+0000","last_peered":"2024-09-15T03:09:17.009761+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:17.009761+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5754,"log_dups_size":0,"ondisk_log_size":5754,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47109746,"num_objects":5751,"num_object_clones":0,"num_object_copies":23004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5752,"num_write_kb":46013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5712","reported_seq":5756,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:16.989872+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:16.989872+0000","last_peered":"2024-09-15T03:09:16.989872+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:16.989872+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46792704,"num_objects":5712,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5712,"num_write_kb":45696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5668","reported_seq":5731,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:16.655765+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:16.655765+0000","last_peered":"2024-09-15T03:09:16.655765+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:16.655765+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5668,"log_dups_size":0,"ondisk_log_size":5668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46380658,"num_objects":5662,"num_object_clones":0,"num_object_copies":22648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5666,"num_write_kb":45308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3610}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5750","reported_seq":5813,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:16.494580+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:16.494580+0000","last_peered":"2024-09-15T03:09:16.494580+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:16.494580+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47060594,"num_objects":5745,"num_object_clones":0,"num_object_copies":22980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5748,"num_write_kb":45972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5679","reported_seq":5736,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:16.987562+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:16.987562+0000","last_peered":"2024-09-15T03:09:16.987562+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:16.987562+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5679,"log_dups_size":0,"ondisk_log_size":5679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46451940,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":45385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3629}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5696","reported_seq":5779,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.593743+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:17.593743+0000","last_peered":"2024-09-15T03:09:17.593743+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:17.593743+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5696,"log_dups_size":0,"ondisk_log_size":5696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46642802,"num_objects":5694,"num_object_clones":0,"num_object_copies":22776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5694,"num_write_kb":45550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3636}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5646","reported_seq":5703,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.102192+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:17.102192+0000","last_peered":"2024-09-15T03:09:17.102192+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:17.102192+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46207403,"num_objects":5641,"num_object_clones":0,"num_object_copies":22564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5643,"num_write_kb":45132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5661","reported_seq":8979,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.055239+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:17.055239+0000","last_peered":"2024-09-15T03:09:17.055239+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:17.055239+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46365497,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5660,"num_write_kb":45279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1641,"num_bytes_recovered":13426688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5307}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5770","reported_seq":5847,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.616272+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:17.616272+0000","last_peered":"2024-09-15T03:09:17.616272+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:17.616272+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5770,"log_dups_size":0,"ondisk_log_size":5770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47258425,"num_objects":5769,"num_object_clones":0,"num_object_copies":23076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5769,"num_write_kb":46151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3730}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5556","reported_seq":5606,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.010690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:17.010690+0000","last_peered":"2024-09-15T03:09:17.010690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:17.010690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5556,"log_dups_size":0,"ondisk_log_size":5556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45479538,"num_objects":5552,"num_object_clones":0,"num_object_copies":22208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5554,"num_write_kb":44428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3516}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5663","reported_seq":5744,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:17.560102+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:17.560102+0000","last_peered":"2024-09-15T03:09:17.560102+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:17.560102+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46364274,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5661,"num_write_kb":45285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3672}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5705","reported_seq":5767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:16.787352+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:16.787352+0000","last_peered":"2024-09-15T03:09:16.787352+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:16.787352+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5705,"log_dups_size":0,"ondisk_log_size":5705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46691954,"num_objects":5700,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5703,"num_write_kb":45612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3618}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":742341180,"num_objects":90622,"num_object_clones":0,"num_object_copies":362488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90649,"num_write_kb":725074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":13426688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1358331904,"data_stored":1358331904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90676,"ondisk_log_size":90676,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668004,"kb_used_data":361820,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93703836,"statfs":{"total":96636764160,"available":95952728064,"internally_reserved":0,"allocated":370503680,"data_stored":370344702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,1,1,5,10,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":3.794},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":3.8860000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":4.3940000000000001},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":4.0209999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":4.1260000000000003},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":3.827}]}]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659456,"kb_used_data":362104,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93712384,"statfs":{"total":96636764160,"available":95961481216,"internally_reserved":0,"allocated":370794496,"data_stored":370638664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,0,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.315}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.376},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":2.6960000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":3.0990000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":536944,"kb_used_data":239080,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93834896,"statfs":{"total":96636764160,"available":96086933504,"internally_reserved":0,"allocated":244817920,"data_stored":244658942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":2.8370000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":12.962999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":11.541}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.42299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660084,"kb_used_data":362284,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93711756,"statfs":{"total":96636764160,"available":95960838144,"internally_reserved":0,"allocated":370978816,"data_stored":370819838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,5,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.40400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.441},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.54700000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":3.9350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":370802688,"data_stored":370802688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":244846592,"data_stored":244846592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371609600,"data_stored":371609600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":371073024,"data_stored":371073024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:22.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 226 20 91907 91887 3.17519 2.00781 0.0398947 0.0491956 2024-09-15T03:09:23.433 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 227 20 92156 92136 3.16977 1.94531 0.164599 0.0492754 2024-09-15T03:09:24.437 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 228 20 92409 92389 3.16454 1.97656 0.133951 0.0493616 2024-09-15T03:09:24.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:25.434 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 229 20 92652 92632 3.15899 1.89844 0.104457 0.0494479 2024-09-15T03:09:26.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:26.254 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:26.278 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-15T03:09:24.738325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":752564796,"num_objects":91870,"num_object_clones":0,"num_object_copies":367480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91897,"num_write_kb":735058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":13828096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91924,"ondisk_log_size":91924,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2566052,"kb_used_data":1346052,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":374921308,"statfs":{"total":386547056640,"available":383919419392,"internally_reserved":0,"allocated":1378357248,"data_stored":1377724482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,8,5,13,13,11,8,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20692992,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":20208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002994"},"pg_stats":[{"pgid":"1.f","version":"31'5740","reported_seq":5782,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.094880+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:22.094880+0000","last_peered":"2024-09-15T03:09:22.094880+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:22.094880+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5740,"log_dups_size":0,"ondisk_log_size":5740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47022080,"num_objects":5740,"num_object_clones":0,"num_object_copies":22960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5515","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.843983+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:21.843983+0000","last_peered":"2024-09-15T03:09:21.843983+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:21.843983+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5515,"log_dups_size":0,"ondisk_log_size":5515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45143666,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5513,"num_write_kb":44100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5894","reported_seq":5975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.598905+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:22.598905+0000","last_peered":"2024-09-15T03:09:22.598905+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:22.598905+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5894,"log_dups_size":0,"ondisk_log_size":5894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48257849,"num_objects":5891,"num_object_clones":0,"num_object_copies":23564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":47134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5586","reported_seq":5649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.806653+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:21.806653+0000","last_peered":"2024-09-15T03:09:21.806653+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:21.806653+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45725298,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5816","reported_seq":5871,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.129130+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:22.129130+0000","last_peered":"2024-09-15T03:09:22.129130+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:22.129130+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5816,"log_dups_size":0,"ondisk_log_size":5816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47617650,"num_objects":5813,"num_object_clones":0,"num_object_copies":23252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5814,"num_write_kb":46509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5803","reported_seq":5847,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.175659+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:22.175659+0000","last_peered":"2024-09-15T03:09:22.175659+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:22.175659+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47538176,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":46424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5755","reported_seq":5820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.698176+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:21.698176+0000","last_peered":"2024-09-15T03:09:21.698176+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:21.698176+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47093362,"num_objects":5749,"num_object_clones":0,"num_object_copies":22996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5753,"num_write_kb":46004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5832","reported_seq":5893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.821039+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:21.821039+0000","last_peered":"2024-09-15T03:09:21.821039+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:21.821039+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47732338,"num_objects":5827,"num_object_clones":0,"num_object_copies":23308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5830,"num_write_kb":46628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5743","reported_seq":5800,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.057208+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:22.057208+0000","last_peered":"2024-09-15T03:09:22.057208+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:22.057208+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46976228,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5785","reported_seq":5868,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.585575+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:22.585575+0000","last_peered":"2024-09-15T03:09:22.585575+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:22.585575+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5785,"log_dups_size":0,"ondisk_log_size":5785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371890,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5783,"num_write_kb":46262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3725}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5722","reported_seq":5780,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.088969+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:22.088969+0000","last_peered":"2024-09-15T03:09:22.088969+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:22.088969+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46829995,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5742","reported_seq":9158,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.024196+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:22.024196+0000","last_peered":"2024-09-15T03:09:22.024196+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:22.024196+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47029049,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":45927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1690,"num_bytes_recovered":13828096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5841","reported_seq":5918,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.674246+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:22.674246+0000","last_peered":"2024-09-15T03:09:22.674246+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:22.674246+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5841,"log_dups_size":0,"ondisk_log_size":5841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47840057,"num_objects":5840,"num_object_clones":0,"num_object_copies":23360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3801}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5622","reported_seq":5672,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.062070+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:22.062070+0000","last_peered":"2024-09-15T03:09:22.062070+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:22.062070+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46020210,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5620,"num_write_kb":44956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3582}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5748","reported_seq":5829,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.616511+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:22.616511+0000","last_peered":"2024-09-15T03:09:22.616511+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:22.616511+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47060594,"num_objects":5745,"num_object_clones":0,"num_object_copies":22980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":45965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5780","reported_seq":5843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.763343+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:21.763343+0000","last_peered":"2024-09-15T03:09:21.763343+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:21.763343+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47306354,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5778,"num_write_kb":46212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":752564796,"num_objects":91870,"num_object_clones":0,"num_object_copies":367480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91897,"num_write_kb":735058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":13828096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1378910208,"data_stored":1378910208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91924,"ondisk_log_size":91924,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677184,"kb_used_data":366840,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93694656,"statfs":{"total":96636764160,"available":95943327744,"internally_reserved":0,"allocated":375644160,"data_stored":375485182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,3,6,1,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.649},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.20499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.217},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":1.609}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.252},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.706}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673808,"kb_used_data":368136,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93698032,"statfs":{"total":96636764160,"available":95946784768,"internally_reserved":0,"allocated":376971264,"data_stored":376815432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2,6,2,2,1,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":3.7549999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":4.2439999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.55000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.435}]}]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546124,"kb_used_data":244100,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93825716,"statfs":{"total":96636764160,"available":96077533184,"internally_reserved":0,"allocated":249958400,"data_stored":249799422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.90600000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":3.5459999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.78300000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":7.8600000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668936,"kb_used_data":366976,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93702904,"statfs":{"total":96636764160,"available":95951773696,"internally_reserved":0,"allocated":375783424,"data_stored":375624446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.71099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.98199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.66900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":375869440,"data_stored":375869440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250138624,"data_stored":250138624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376717312,"data_stored":376717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376184832,"data_stored":376184832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:26.279 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:26.434 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 230 20 92891 92871 3.15338 1.86719 0.0705536 0.0495368 2024-09-15T03:09:27.434 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 231 20 93141 93121 3.14818 1.95312 0.0924681 0.049616 2024-09-15T03:09:27.680 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:27.712 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-15T03:09:26.743652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":752564796,"num_objects":91870,"num_object_clones":0,"num_object_copies":367480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91897,"num_write_kb":735058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":13828096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91924,"ondisk_log_size":91924,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2566052,"kb_used_data":1346052,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":374921308,"statfs":{"total":386547056640,"available":383919419392,"internally_reserved":0,"allocated":1378357248,"data_stored":1377724482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,8,5,13,13,11,8,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20692992,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2526,"num_write_kb":20208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007792"},"pg_stats":[{"pgid":"1.f","version":"31'5740","reported_seq":5782,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.094880+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:22.094880+0000","last_peered":"2024-09-15T03:09:22.094880+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:22.094880+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5740,"log_dups_size":0,"ondisk_log_size":5740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47022080,"num_objects":5740,"num_object_clones":0,"num_object_copies":22960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5515","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.843983+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:21.843983+0000","last_peered":"2024-09-15T03:09:21.843983+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:21.843983+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5515,"log_dups_size":0,"ondisk_log_size":5515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45143666,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5513,"num_write_kb":44100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5894","reported_seq":5975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.598905+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:22.598905+0000","last_peered":"2024-09-15T03:09:22.598905+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:22.598905+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5894,"log_dups_size":0,"ondisk_log_size":5894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48257849,"num_objects":5891,"num_object_clones":0,"num_object_copies":23564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":47134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3789}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5586","reported_seq":5649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.806653+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:21.806653+0000","last_peered":"2024-09-15T03:09:21.806653+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:21.806653+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45725298,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5816","reported_seq":5871,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.129130+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:22.129130+0000","last_peered":"2024-09-15T03:09:22.129130+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:22.129130+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5816,"log_dups_size":0,"ondisk_log_size":5816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47617650,"num_objects":5813,"num_object_clones":0,"num_object_copies":23252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5814,"num_write_kb":46509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5803","reported_seq":5847,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.175659+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:22.175659+0000","last_peered":"2024-09-15T03:09:22.175659+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:22.175659+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47538176,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":46424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5755","reported_seq":5820,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.698176+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:21.698176+0000","last_peered":"2024-09-15T03:09:21.698176+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:21.698176+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47093362,"num_objects":5749,"num_object_clones":0,"num_object_copies":22996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5753,"num_write_kb":46004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5832","reported_seq":5893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.821039+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:21.821039+0000","last_peered":"2024-09-15T03:09:21.821039+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:21.821039+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47732338,"num_objects":5827,"num_object_clones":0,"num_object_copies":23308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5830,"num_write_kb":46628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5743","reported_seq":5800,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.057208+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:22.057208+0000","last_peered":"2024-09-15T03:09:22.057208+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:22.057208+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46976228,"num_objects":5735,"num_object_clones":0,"num_object_copies":22940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5785","reported_seq":5868,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.585575+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:22.585575+0000","last_peered":"2024-09-15T03:09:22.585575+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:22.585575+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5785,"log_dups_size":0,"ondisk_log_size":5785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371890,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5783,"num_write_kb":46262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3725}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5722","reported_seq":5780,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.088969+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:22.088969+0000","last_peered":"2024-09-15T03:09:22.088969+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:22.088969+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5722,"log_dups_size":0,"ondisk_log_size":5722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46829995,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3639}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5742","reported_seq":9158,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.024196+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:22.024196+0000","last_peered":"2024-09-15T03:09:22.024196+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:22.024196+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47029049,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":45927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1690,"num_bytes_recovered":13828096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5437}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5841","reported_seq":5918,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.674246+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:22.674246+0000","last_peered":"2024-09-15T03:09:22.674246+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:22.674246+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5841,"log_dups_size":0,"ondisk_log_size":5841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47840057,"num_objects":5840,"num_object_clones":0,"num_object_copies":23360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3801}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5622","reported_seq":5672,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.062070+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:22.062070+0000","last_peered":"2024-09-15T03:09:22.062070+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:22.062070+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5622,"log_dups_size":0,"ondisk_log_size":5622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46020210,"num_objects":5618,"num_object_clones":0,"num_object_copies":22472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5620,"num_write_kb":44956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3582}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5748","reported_seq":5829,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:22.616511+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:22.616511+0000","last_peered":"2024-09-15T03:09:22.616511+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:22.616511+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5748,"log_dups_size":0,"ondisk_log_size":5748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47060594,"num_objects":5745,"num_object_clones":0,"num_object_copies":22980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":45965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5780","reported_seq":5843,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:21.763343+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:21.763343+0000","last_peered":"2024-09-15T03:09:21.763343+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:21.763343+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47306354,"num_objects":5775,"num_object_clones":0,"num_object_copies":23100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5778,"num_write_kb":46212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":752564796,"num_objects":91870,"num_object_clones":0,"num_object_copies":367480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91897,"num_write_kb":735058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":13828096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1378910208,"data_stored":1378910208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91924,"ondisk_log_size":91924,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677184,"kb_used_data":366840,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93694656,"statfs":{"total":96636764160,"available":95943327744,"internally_reserved":0,"allocated":375644160,"data_stored":375485182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,3,6,1,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.649},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.20499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.217},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":1.609}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":0.252},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":1.706}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673808,"kb_used_data":368136,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93698032,"statfs":{"total":96636764160,"available":95946784768,"internally_reserved":0,"allocated":376971264,"data_stored":376815432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2,6,2,2,1,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:21 2024","interfaces":[{"interface":"back","average":{"1min":2.1379999999999999,"5min":1.5309999999999999,"15min":0.99099999999999999},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":8.3759999999999994,"5min":23.280000000000001,"15min":23.280000000000001},"last":3.7549999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.1659999999999999,"15min":0.93600000000000005},"min":{"1min":0.22,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":9.1050000000000004,"5min":19.172999999999998,"15min":19.172999999999998},"last":4.2439999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.55000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.435}]}]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546124,"kb_used_data":244100,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93825716,"statfs":{"total":96636764160,"available":96077533184,"internally_reserved":0,"allocated":249958400,"data_stored":249799422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.90600000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":3.5459999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.78300000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":7.8600000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668936,"kb_used_data":366976,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93702904,"statfs":{"total":96636764160,"available":95951773696,"internally_reserved":0,"allocated":375783424,"data_stored":375624446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.71099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":0.98199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.66900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":375869440,"data_stored":375869440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":250138624,"data_stored":250138624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376717312,"data_stored":376717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376184832,"data_stored":376184832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:27.730 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:09:27.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5740", 'reported_seq': 5782, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.094880+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:09:22.094880+0000', 'last_peered': '2024-09-15T03:09:22.094880+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:09:22.094880+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5740, 'log_dups_size': 0, 'ondisk_log_size': 5740, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47022080, 'num_objects': 5740, 'num_object_clones': 0, 'num_object_copies': 22960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5740, 'num_write_kb': 45920, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3764}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:27.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:09:27.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5515", 'reported_seq': 5577, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:21.843983+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:09:21.843983+0000', 'last_peered': '2024-09-15T03:09:21.843983+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:09:21.843983+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5515, 'log_dups_size': 0, 'ondisk_log_size': 5515, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45143666, 'num_objects': 5511, 'num_object_clones': 0, 'num_object_copies': 22044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5513, 'num_write_kb': 44100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:27.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:09:27.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5894", 'reported_seq': 5975, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.598905+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:09:22.598905+0000', 'last_peered': '2024-09-15T03:09:22.598905+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:09:22.598905+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5894, 'log_dups_size': 0, 'ondisk_log_size': 5894, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48257849, 'num_objects': 5891, 'num_object_clones': 0, 'num_object_copies': 23564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5893, 'num_write_kb': 47134, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3789}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5586", 'reported_seq': 5649, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:21.806653+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:09:21.806653+0000', 'last_peered': '2024-09-15T03:09:21.806653+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:09:21.806653+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5586, 'log_dups_size': 0, 'ondisk_log_size': 5586, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45725298, 'num_objects': 5582, 'num_object_clones': 0, 'num_object_copies': 22328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5584, 'num_write_kb': 44661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5816", 'reported_seq': 5871, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.129130+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:09:22.129130+0000', 'last_peered': '2024-09-15T03:09:22.129130+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:09:22.129130+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5816, 'log_dups_size': 0, 'ondisk_log_size': 5816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47617650, 'num_objects': 5813, 'num_object_clones': 0, 'num_object_copies': 23252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5814, 'num_write_kb': 46509, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3706}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5803", 'reported_seq': 5847, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.175659+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:09:22.175659+0000', 'last_peered': '2024-09-15T03:09:22.175659+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:09:22.175659+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5803, 'log_dups_size': 0, 'ondisk_log_size': 5803, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47538176, 'num_objects': 5803, 'num_object_clones': 0, 'num_object_copies': 23212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5803, 'num_write_kb': 46424, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3747}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5755", 'reported_seq': 5820, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:21.698176+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:09:21.698176+0000', 'last_peered': '2024-09-15T03:09:21.698176+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:09:21.698176+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5755, 'log_dups_size': 0, 'ondisk_log_size': 5755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47093362, 'num_objects': 5749, 'num_object_clones': 0, 'num_object_copies': 22996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5753, 'num_write_kb': 46004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3697}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:27.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5832", 'reported_seq': 5893, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:21.821039+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:09:21.821039+0000', 'last_peered': '2024-09-15T03:09:21.821039+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:09:21.821039+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5832, 'log_dups_size': 0, 'ondisk_log_size': 5832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47732338, 'num_objects': 5827, 'num_object_clones': 0, 'num_object_copies': 23308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5830, 'num_write_kb': 46628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3746}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5743", 'reported_seq': 5800, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.057208+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:09:22.057208+0000', 'last_peered': '2024-09-15T03:09:22.057208+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:09:22.057208+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5743, 'log_dups_size': 0, 'ondisk_log_size': 5743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46976228, 'num_objects': 5735, 'num_object_clones': 0, 'num_object_copies': 22940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5740, 'num_write_kb': 45897, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3693}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5785", 'reported_seq': 5868, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.585575+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:09:22.585575+0000', 'last_peered': '2024-09-15T03:09:22.585575+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:09:22.585575+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5785, 'log_dups_size': 0, 'ondisk_log_size': 5785, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47371890, 'num_objects': 5783, 'num_object_clones': 0, 'num_object_copies': 23132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5783, 'num_write_kb': 46262, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3725}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5722", 'reported_seq': 5780, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.088969+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:09:22.088969+0000', 'last_peered': '2024-09-15T03:09:22.088969+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:09:22.088969+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5722, 'log_dups_size': 0, 'ondisk_log_size': 5722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46829995, 'num_objects': 5717, 'num_object_clones': 0, 'num_object_copies': 22868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5719, 'num_write_kb': 45740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3639}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5742", 'reported_seq': 9158, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.024196+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:09:22.024196+0000', 'last_peered': '2024-09-15T03:09:22.024196+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:09:22.024196+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5742, 'log_dups_size': 0, 'ondisk_log_size': 5742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47029049, 'num_objects': 5741, 'num_object_clones': 0, 'num_object_copies': 22964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5741, 'num_write_kb': 45927, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1690, 'num_bytes_recovered': 13828096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5437}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:09:27.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5841", 'reported_seq': 5918, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.674246+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:09:22.674246+0000', 'last_peered': '2024-09-15T03:09:22.674246+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:09:22.674246+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5841, 'log_dups_size': 0, 'ondisk_log_size': 5841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47840057, 'num_objects': 5840, 'num_object_clones': 0, 'num_object_copies': 23360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5840, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5840, 'num_write_kb': 46719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3801}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:27.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:09:27.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5622", 'reported_seq': 5672, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.062070+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:09:22.062070+0000', 'last_peered': '2024-09-15T03:09:22.062070+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:09:22.062070+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5622, 'log_dups_size': 0, 'ondisk_log_size': 5622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46020210, 'num_objects': 5618, 'num_object_clones': 0, 'num_object_copies': 22472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5618, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5620, 'num_write_kb': 44956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3582}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:27.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:09:27.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5748", 'reported_seq': 5829, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:22.616511+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:09:22.616511+0000', 'last_peered': '2024-09-15T03:09:22.616511+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:09:22.616511+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5748, 'log_dups_size': 0, 'ondisk_log_size': 5748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47060594, 'num_objects': 5745, 'num_object_clones': 0, 'num_object_copies': 22980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5746, 'num_write_kb': 45965, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3757}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:27.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:09:27.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5780", 'reported_seq': 5843, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:21.763343+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:09:21.763343+0000', 'last_peered': '2024-09-15T03:09:21.763343+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:09:21.763343+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5780, 'log_dups_size': 0, 'ondisk_log_size': 5780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47306354, 'num_objects': 5775, 'num_object_clones': 0, 'num_object_copies': 23100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5778, 'num_write_kb': 46212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3693}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:27.734 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:09:28.435 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 232 20 93378 93358 3.1426 1.85156 0.152022 0.0497028 2024-09-15T03:09:29.437 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 233 20 93627 93607 3.13743 1.94531 0.0682343 0.0497864 2024-09-15T03:09:29.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:29.736 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30893/372448 objects degraded (8.295%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93112,"data_bytes":762739260,"bytes_used":2655838208,"bytes_avail":383891218432,"bytes_total":386547056640,"degraded_objects":30893,"degraded_total":372448,"degraded_ratio":0.082945807199931262,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":122117,"recovering_keys_per_sec":0,"num_objects_recovered":179,"num_bytes_recovered":1466368,"num_keys_recovered":0,"write_bytes_sec":2570612,"read_op_per_sec":0,"write_op_per_sec":313},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:09:29.829 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:09:29.829 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:30.437 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 234 20 93866 93846 3.132 1.86719 0.0788259 0.049876 2024-09-15T03:09:31.437 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 235 20 94104 94084 3.12659 1.85938 0.0490157 0.0499595 2024-09-15T03:09:31.582 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:31.608 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:31.674 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-15T03:09:30.744688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":762739260,"num_objects":93112,"num_object_clones":0,"num_object_copies":372448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93139,"num_write_kb":744994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1791,"num_bytes_recovered":14442496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93166,"ondisk_log_size":93166,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2593592,"kb_used_data":1365272,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":374893768,"statfs":{"total":386547056640,"available":383891218432,"internally_reserved":0,"allocated":1398038528,"data_stored":1397405762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,9,14,11,13,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20398080,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":19920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":1015808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007911"},"pg_stats":[{"pgid":"1.f","version":"31'5820","reported_seq":5858,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.220855+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:27.220855+0000","last_peered":"2024-09-15T03:09:27.220855+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:27.220855+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5820,"log_dups_size":0,"ondisk_log_size":5820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47677440,"num_objects":5820,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5820,"num_write_kb":46560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5587","reported_seq":5647,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:26.879512+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:26.879512+0000","last_peered":"2024-09-15T03:09:26.879512+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:26.879512+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45733490,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5585,"num_write_kb":44676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5983","reported_seq":6064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.611321+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:27.611321+0000","last_peered":"2024-09-15T03:09:27.611321+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:27.611321+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5983,"log_dups_size":0,"ondisk_log_size":5983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48986937,"num_objects":5980,"num_object_clones":0,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5982,"num_write_kb":47846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3878}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5660","reported_seq":5723,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:26.900955+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:26.900955+0000","last_peered":"2024-09-15T03:09:26.900955+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:26.900955+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46331506,"num_objects":5656,"num_object_clones":0,"num_object_copies":22624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5890","reported_seq":5946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.205010+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:27.205010+0000","last_peered":"2024-09-15T03:09:27.205010+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:27.205010+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5890,"log_dups_size":0,"ondisk_log_size":5890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48223858,"num_objects":5887,"num_object_clones":0,"num_object_copies":23548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5888,"num_write_kb":47101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5869","reported_seq":5913,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.014527+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:27.014527+0000","last_peered":"2024-09-15T03:09:27.014527+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:27.014527+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5869,"log_dups_size":0,"ondisk_log_size":5869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48078848,"num_objects":5869,"num_object_clones":0,"num_object_copies":23476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5869,"num_write_kb":46952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5836","reported_seq":5901,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:26.790903+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:26.790903+0000","last_peered":"2024-09-15T03:09:26.790903+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:26.790903+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47756914,"num_objects":5830,"num_object_clones":0,"num_object_copies":23320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5834,"num_write_kb":46652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3778}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5914","reported_seq":5976,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:26.838145+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:26.838145+0000","last_peered":"2024-09-15T03:09:26.838145+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:26.838145+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5914,"log_dups_size":0,"ondisk_log_size":5914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48404082,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5912,"num_write_kb":47284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5828","reported_seq":5885,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.217386+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:27.217386+0000","last_peered":"2024-09-15T03:09:27.217386+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:27.217386+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47672548,"num_objects":5820,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5825,"num_write_kb":46577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5857","reported_seq":5940,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.474987+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:27.474987+0000","last_peered":"2024-09-15T03:09:27.474987+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:27.474987+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47961714,"num_objects":5855,"num_object_clones":0,"num_object_copies":23420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5855,"num_write_kb":46838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3797}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5806","reported_seq":5864,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.131175+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:27.131175+0000","last_peered":"2024-09-15T03:09:27.131175+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:27.131175+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47518123,"num_objects":5801,"num_object_clones":0,"num_object_copies":23204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5803,"num_write_kb":46412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5813","reported_seq":9379,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.224274+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:27.224274+0000","last_peered":"2024-09-15T03:09:27.224274+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:27.224274+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47610681,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5812,"num_write_kb":46495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1765,"num_bytes_recovered":14442496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5583}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5923","reported_seq":6000,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.598340+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:27.598340+0000","last_peered":"2024-09-15T03:09:27.598340+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:27.598340+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5923,"log_dups_size":0,"ondisk_log_size":5923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48511801,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5922,"num_write_kb":47375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5703","reported_seq":5753,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:26.912762+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:26.912762+0000","last_peered":"2024-09-15T03:09:26.912762+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:26.912762+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46683762,"num_objects":5699,"num_object_clones":0,"num_object_copies":22796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5701,"num_write_kb":45604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3663}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5822","reported_seq":5903,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:27.580740+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:27.580740+0000","last_peered":"2024-09-15T03:09:27.580740+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:27.580740+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5822,"log_dups_size":0,"ondisk_log_size":5822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47666802,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5820,"num_write_kb":46557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3831}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5855","reported_seq":5918,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:26.860471+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:26.860471+0000","last_peered":"2024-09-15T03:09:26.860471+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:26.860471+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47920754,"num_objects":5850,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5853,"num_write_kb":46812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3768}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":762739260,"num_objects":93112,"num_object_clones":0,"num_object_copies":372448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93139,"num_write_kb":744994,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1791,"num_bytes_recovered":14442496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1399590912,"data_stored":1399590912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93166,"ondisk_log_size":93166,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673792,"kb_used_data":371768,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93698048,"statfs":{"total":96636764160,"available":95946801152,"internally_reserved":0,"allocated":380690432,"data_stored":380531454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,4,1,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.57799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":1.5},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.26400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":8.0739999999999998},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":0.41699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681944,"kb_used_data":372112,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93689896,"statfs":{"total":96636764160,"available":95938453504,"internally_reserved":0,"allocated":381042688,"data_stored":380886856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.7879999999999998},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.2450000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.44800000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.4019999999999999,"15min":0.87},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":8.2810000000000006,"5min":21.600999999999999,"15min":21.600999999999999},"last":6.1630000000000003},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.427,"15min":0.98799999999999999},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.231,"5min":19.274000000000001,"15min":19.274000000000001},"last":6.1210000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559772,"kb_used_data":249428,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93812068,"statfs":{"total":96636764160,"available":96063557632,"internally_reserved":0,"allocated":255414272,"data_stored":255255294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":20.791}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":3.0150000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.35799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.2549999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":2.2999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678084,"kb_used_data":371964,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93693756,"statfs":{"total":96636764160,"available":95942406144,"internally_reserved":0,"allocated":380891136,"data_stored":380732158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.52200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:08:29 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.714,"15min":1.194},"min":{"1min":0.20999999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.802,"5min":30.802,"15min":30.802},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.92000000000000004,"15min":0.71299999999999997},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0550000000000002,"5min":8.1159999999999997,"15min":8.1159999999999997},"last":3.2589999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.48399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":380960768,"data_stored":380960768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255594496,"data_stored":255594496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":381734912,"data_stored":381734912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":381300736,"data_stored":381300736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:32.437 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 236 20 94353 94333 3.12158 1.94531 0.176959 0.0500373 2024-09-15T03:09:33.437 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 237 20 94603 94583 3.11665 1.95312 0.0344343 0.0501123 2024-09-15T03:09:34.437 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 238 20 94857 94837 3.11189 1.98438 0.0663495 0.0501969 2024-09-15T03:09:34.677 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:35.437 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout: 239 20 95093 95073 3.10659 1.84375 0.159204 0.0502767 2024-09-15T03:09:35.811 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:35.825 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:35.846 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-15T03:09:34.745741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":772831804,"num_objects":94344,"num_object_clones":0,"num_object_copies":377376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94371,"num_write_kb":754850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1879,"num_bytes_recovered":15163392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94398,"ondisk_log_size":94398,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2600964,"kb_used_data":1385124,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":374886396,"statfs":{"total":386547056640,"available":383883669504,"internally_reserved":0,"allocated":1418366976,"data_stored":1417734210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,5,9,20,15,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20267008,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2474,"num_write_kb":19792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":1335296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008003"},"pg_stats":[{"pgid":"1.f","version":"31'5883","reported_seq":5925,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.067077+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:32.067077+0000","last_peered":"2024-09-15T03:09:32.067077+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:32.067077+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48193536,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5883,"num_write_kb":47064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5653","reported_seq":5712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.824165+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:31.824165+0000","last_peered":"2024-09-15T03:09:31.824165+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:31.824165+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46274162,"num_objects":5649,"num_object_clones":0,"num_object_copies":22596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":45204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'6058","reported_seq":6137,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.737767+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:32.737767+0000","last_peered":"2024-09-15T03:09:32.737767+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:32.737767+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6058,"log_dups_size":0,"ondisk_log_size":6058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49601337,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6057,"num_write_kb":48446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3953}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5742","reported_seq":5804,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.730746+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:31.730746+0000","last_peered":"2024-09-15T03:09:31.730746+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:31.730746+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47003250,"num_objects":5738,"num_object_clones":0,"num_object_copies":22952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5981","reported_seq":6037,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.267764+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:32.267764+0000","last_peered":"2024-09-15T03:09:32.267764+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:32.267764+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5981,"log_dups_size":0,"ondisk_log_size":5981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48969330,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":47829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5947","reported_seq":5991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.262985+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:32.262985+0000","last_peered":"2024-09-15T03:09:32.262985+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:32.262985+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5947,"log_dups_size":0,"ondisk_log_size":5947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48717824,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5947,"num_write_kb":47576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5917","reported_seq":5981,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.867950+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:31.867950+0000","last_peered":"2024-09-15T03:09:31.867950+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:31.867950+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48420466,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5990","reported_seq":6053,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.894575+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:31.894575+0000","last_peered":"2024-09-15T03:09:31.894575+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:31.894575+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5990,"log_dups_size":0,"ondisk_log_size":5990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49026674,"num_objects":5985,"num_object_clones":0,"num_object_copies":23940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3904}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5918","reported_seq":5975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.211627+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:32.211627+0000","last_peered":"2024-09-15T03:09:32.211627+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:32.211627+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48409828,"num_objects":5910,"num_object_clones":0,"num_object_copies":23640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5928","reported_seq":6011,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.627096+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:32.627096+0000","last_peered":"2024-09-15T03:09:32.627096+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:32.627096+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48543346,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":47406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3868}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5882","reported_seq":5940,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.273367+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:32.273367+0000","last_peered":"2024-09-15T03:09:32.273367+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:32.273367+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5882,"log_dups_size":0,"ondisk_log_size":5882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48140715,"num_objects":5877,"num_object_clones":0,"num_object_copies":23508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5888","reported_seq":9630,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.261027+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:32.261027+0000","last_peered":"2024-09-15T03:09:32.261027+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:32.261027+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48225081,"num_objects":5887,"num_object_clones":0,"num_object_copies":23548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5887,"num_write_kb":47095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":15163392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5746}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6004","reported_seq":6081,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.707829+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:32.707829+0000","last_peered":"2024-09-15T03:09:32.707829+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:32.707829+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6004,"log_dups_size":0,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49175353,"num_objects":6003,"num_object_clones":0,"num_object_copies":24012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6003,"num_write_kb":48023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3964}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5780","reported_seq":5830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.127615+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:32.127615+0000","last_peered":"2024-09-15T03:09:32.127615+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:32.127615+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314546,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5778,"num_write_kb":46220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3740}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5895","reported_seq":5976,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.712813+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:32.712813+0000","last_peered":"2024-09-15T03:09:32.712813+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:32.712813+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5895,"log_dups_size":0,"ondisk_log_size":5895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48264818,"num_objects":5892,"num_object_clones":0,"num_object_copies":23568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":47141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3904}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5932","reported_seq":5995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.805096+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:31.805096+0000","last_peered":"2024-09-15T03:09:31.805096+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:31.805096+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48551538,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":47428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":772831804,"num_objects":94344,"num_object_clones":0,"num_object_copies":377376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94371,"num_write_kb":754850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1879,"num_bytes_recovered":15163392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1420120064,"data_stored":1420120064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94398,"ondisk_log_size":94398,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682884,"kb_used_data":376700,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93688956,"statfs":{"total":96636764160,"available":95937490944,"internally_reserved":0,"allocated":385740800,"data_stored":385581822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,11,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":13.266999999999999},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":13.052}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":7.4560000000000004},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":13.343999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":4.0430000000000001},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":13.308999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674328,"kb_used_data":376976,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93697512,"statfs":{"total":96636764160,"available":95946252288,"internally_reserved":0,"allocated":386023424,"data_stored":385867592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.47899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.39600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.438},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.63600000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552468,"kb_used_data":254604,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93819372,"statfs":{"total":96636764160,"available":96071036928,"internally_reserved":0,"allocated":260714496,"data_stored":260555518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.28000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":12.108000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.60499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":7.2999999999999998},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.40200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691284,"kb_used_data":376844,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93680556,"statfs":{"total":96636764160,"available":95928889344,"internally_reserved":0,"allocated":385888256,"data_stored":385729278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,4,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":7.4299999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":7.391},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.45600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":7.2039999999999997},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385953792,"data_stored":385953792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261070848,"data_stored":261070848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386797568,"data_stored":386797568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":386297856,"data_stored":386297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:35.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:36.438 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Total time run: 240.07 2024-09-15T03:09:36.438 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Total writes made: 95311 2024-09-15T03:09:36.438 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Write size: 8192 2024-09-15T03:09:36.438 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Object size: 8192 2024-09-15T03:09:36.439 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Bandwidth (MB/sec): 3.10167 2024-09-15T03:09:36.439 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Stddev Bandwidth: 2.16901 2024-09-15T03:09:36.439 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Max bandwidth (MB/sec): 12.7891 2024-09-15T03:09:36.439 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Min bandwidth (MB/sec): 0 2024-09-15T03:09:36.439 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Average IOPS: 397 2024-09-15T03:09:36.439 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Stddev IOPS: 277.636 2024-09-15T03:09:36.440 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Max IOPS: 1637 2024-09-15T03:09:36.440 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Min IOPS: 0 2024-09-15T03:09:36.440 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Average Latency(s): 0.0503655 2024-09-15T03:09:36.440 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Stddev Latency(s): 0.0540454 2024-09-15T03:09:36.440 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Max latency(s): 2.51529 2024-09-15T03:09:36.440 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Min latency(s): 0.00240791 2024-09-15T03:09:36.454 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Cleaning up (deleting benchmark objects) 2024-09-15T03:09:36.985 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:37.008 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:37.037 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-15T03:09:36.746239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":772831804,"num_objects":94344,"num_object_clones":0,"num_object_copies":377376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94371,"num_write_kb":754850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1879,"num_bytes_recovered":15163392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94398,"ondisk_log_size":94398,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2600964,"kb_used_data":1385124,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":374886396,"statfs":{"total":386547056640,"available":383883669504,"internally_reserved":0,"allocated":1418366976,"data_stored":1417734210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,5,9,20,15,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20267008,"num_objects":2474,"num_object_clones":0,"num_object_copies":9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2474,"num_write_kb":19792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":1335296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007913"},"pg_stats":[{"pgid":"1.f","version":"31'5883","reported_seq":5925,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.067077+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:32.067077+0000","last_peered":"2024-09-15T03:09:32.067077+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:32.067077+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48193536,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5883,"num_write_kb":47064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5653","reported_seq":5712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.824165+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:31.824165+0000","last_peered":"2024-09-15T03:09:31.824165+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:31.824165+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46274162,"num_objects":5649,"num_object_clones":0,"num_object_copies":22596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5651,"num_write_kb":45204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'6058","reported_seq":6137,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.737767+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:32.737767+0000","last_peered":"2024-09-15T03:09:32.737767+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:32.737767+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6058,"log_dups_size":0,"ondisk_log_size":6058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49601337,"num_objects":6055,"num_object_clones":0,"num_object_copies":24220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6057,"num_write_kb":48446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3953}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5742","reported_seq":5804,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.730746+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:31.730746+0000","last_peered":"2024-09-15T03:09:31.730746+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:31.730746+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47003250,"num_objects":5738,"num_object_clones":0,"num_object_copies":22952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5740,"num_write_kb":45909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3795}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5981","reported_seq":6037,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.267764+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:32.267764+0000","last_peered":"2024-09-15T03:09:32.267764+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:32.267764+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5981,"log_dups_size":0,"ondisk_log_size":5981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48969330,"num_objects":5978,"num_object_clones":0,"num_object_copies":23912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":47829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3871}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5947","reported_seq":5991,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.262985+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:32.262985+0000","last_peered":"2024-09-15T03:09:32.262985+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:32.262985+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5947,"log_dups_size":0,"ondisk_log_size":5947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48717824,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5947,"num_write_kb":47576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3891}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5917","reported_seq":5981,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.867950+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:31.867950+0000","last_peered":"2024-09-15T03:09:31.867950+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:31.867950+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48420466,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5990","reported_seq":6053,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.894575+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:31.894575+0000","last_peered":"2024-09-15T03:09:31.894575+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:31.894575+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5990,"log_dups_size":0,"ondisk_log_size":5990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49026674,"num_objects":5985,"num_object_clones":0,"num_object_copies":23940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3904}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5918","reported_seq":5975,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.211627+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:32.211627+0000","last_peered":"2024-09-15T03:09:32.211627+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:32.211627+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48409828,"num_objects":5910,"num_object_clones":0,"num_object_copies":23640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5928","reported_seq":6011,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.627096+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:32.627096+0000","last_peered":"2024-09-15T03:09:32.627096+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:32.627096+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48543346,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":47406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3868}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5882","reported_seq":5940,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.273367+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:32.273367+0000","last_peered":"2024-09-15T03:09:32.273367+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:32.273367+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5882,"log_dups_size":0,"ondisk_log_size":5882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48140715,"num_objects":5877,"num_object_clones":0,"num_object_copies":23508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5879,"num_write_kb":47020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5888","reported_seq":9630,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.261027+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:32.261027+0000","last_peered":"2024-09-15T03:09:32.261027+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:32.261027+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48225081,"num_objects":5887,"num_object_clones":0,"num_object_copies":23548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5887,"num_write_kb":47095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":15163392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5746}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6004","reported_seq":6081,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.707829+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:32.707829+0000","last_peered":"2024-09-15T03:09:32.707829+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:32.707829+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6004,"log_dups_size":0,"ondisk_log_size":6004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49175353,"num_objects":6003,"num_object_clones":0,"num_object_copies":24012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6003,"num_write_kb":48023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3964}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5780","reported_seq":5830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.127615+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:32.127615+0000","last_peered":"2024-09-15T03:09:32.127615+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:32.127615+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5780,"log_dups_size":0,"ondisk_log_size":5780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314546,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5778,"num_write_kb":46220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3740}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5895","reported_seq":5976,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:32.712813+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:32.712813+0000","last_peered":"2024-09-15T03:09:32.712813+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:32.712813+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5895,"log_dups_size":0,"ondisk_log_size":5895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48264818,"num_objects":5892,"num_object_clones":0,"num_object_copies":23568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5893,"num_write_kb":47141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3904}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5932","reported_seq":5995,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:31.805096+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:31.805096+0000","last_peered":"2024-09-15T03:09:31.805096+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:31.805096+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5932,"log_dups_size":0,"ondisk_log_size":5932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48551538,"num_objects":5927,"num_object_clones":0,"num_object_copies":23708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5930,"num_write_kb":47428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":772831804,"num_objects":94344,"num_object_clones":0,"num_object_copies":377376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94371,"num_write_kb":754850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1879,"num_bytes_recovered":15163392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1420120064,"data_stored":1420120064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94398,"ondisk_log_size":94398,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682884,"kb_used_data":376700,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93688956,"statfs":{"total":96636764160,"available":95937490944,"internally_reserved":0,"allocated":385740800,"data_stored":385581822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,11,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.738,"15min":1.708},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.8780000000000001,"5min":17.408999999999999,"15min":17.408999999999999},"last":13.266999999999999},{"interface":"front","average":{"1min":3.0699999999999998,"5min":2.2210000000000001,"15min":2.1179999999999999},"min":{"1min":0.193,"5min":0.188,"15min":0.188},"max":{"1min":16.783999999999999,"5min":17.542000000000002,"15min":17.542000000000002},"last":13.052}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":7.4560000000000004},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":13.343999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:08:34 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.3049999999999999,"15min":0.78900000000000003},"min":{"1min":0.25700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":16.216999999999999,"5min":16.216999999999999,"15min":16.216999999999999},"last":4.0430000000000001},{"interface":"front","average":{"1min":1.661,"5min":1.173,"15min":1.0149999999999999},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.231999999999999,"5min":12.231999999999999,"15min":12.231999999999999},"last":13.308999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674328,"kb_used_data":376976,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93697512,"statfs":{"total":96636764160,"available":95946252288,"internally_reserved":0,"allocated":386023424,"data_stored":385867592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.47899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.39600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.438},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.63600000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552468,"kb_used_data":254604,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93819372,"statfs":{"total":96636764160,"available":96071036928,"internally_reserved":0,"allocated":260714496,"data_stored":260555518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.28000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":12.108000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.60499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":7.2999999999999998},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.40200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691284,"kb_used_data":376844,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93680556,"statfs":{"total":96636764160,"available":95928889344,"internally_reserved":0,"allocated":385888256,"data_stored":385729278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,4,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":7.4299999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":7.391},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.45600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:08:33 2024","interfaces":[{"interface":"back","average":{"1min":2.4279999999999999,"5min":1.464,"15min":0.92800000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.948,"5min":20.948,"15min":20.948},"last":7.2039999999999997},{"interface":"front","average":{"1min":1.726,"5min":0.98299999999999998,"15min":0.64000000000000001},"min":{"1min":0.25600000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5309999999999997,"5min":11.6,"15min":11.6},"last":0.311}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":385953792,"data_stored":385953792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261070848,"data_stored":261070848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386797568,"data_stored":386797568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":386297856,"data_stored":386297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:37.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:09:37.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5883", 'reported_seq': 5925, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.067077+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:09:32.067077+0000', 'last_peered': '2024-09-15T03:09:32.067077+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:09:32.067077+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5883, 'log_dups_size': 0, 'ondisk_log_size': 5883, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48193536, 'num_objects': 5883, 'num_object_clones': 0, 'num_object_copies': 23532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5883, 'num_write_kb': 47064, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3907}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:37.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:09:37.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5653", 'reported_seq': 5712, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:31.824165+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:09:31.824165+0000', 'last_peered': '2024-09-15T03:09:31.824165+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:09:31.824165+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5653, 'log_dups_size': 0, 'ondisk_log_size': 5653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46274162, 'num_objects': 5649, 'num_object_clones': 0, 'num_object_copies': 22596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5651, 'num_write_kb': 45204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:37.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:09:37.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'6058", 'reported_seq': 6137, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.737767+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:09:32.737767+0000', 'last_peered': '2024-09-15T03:09:32.737767+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:09:32.737767+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6058, 'log_dups_size': 0, 'ondisk_log_size': 6058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49601337, 'num_objects': 6055, 'num_object_clones': 0, 'num_object_copies': 24220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6057, 'num_write_kb': 48446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3953}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:37.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:09:37.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5742", 'reported_seq': 5804, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:31.730746+0000', 'last_change': '2024-09-15T03:06:09.201047+0000', 'last_active': '2024-09-15T03:09:31.730746+0000', 'last_peered': '2024-09-15T03:09:31.730746+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:09:31.730746+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5742, 'log_dups_size': 0, 'ondisk_log_size': 5742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47003250, 'num_objects': 5738, 'num_object_clones': 0, 'num_object_copies': 22952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1945, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5740, 'num_write_kb': 45909, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5981", 'reported_seq': 6037, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.267764+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:09:32.267764+0000', 'last_peered': '2024-09-15T03:09:32.267764+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:09:32.267764+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5981, 'log_dups_size': 0, 'ondisk_log_size': 5981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48969330, 'num_objects': 5978, 'num_object_clones': 0, 'num_object_copies': 23912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5979, 'num_write_kb': 47829, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3871}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5947", 'reported_seq': 5991, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.262985+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:09:32.262985+0000', 'last_peered': '2024-09-15T03:09:32.262985+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:09:32.262985+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5947, 'log_dups_size': 0, 'ondisk_log_size': 5947, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48717824, 'num_objects': 5947, 'num_object_clones': 0, 'num_object_copies': 23788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5947, 'num_write_kb': 47576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3891}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5917", 'reported_seq': 5981, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:31.867950+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:09:31.867950+0000', 'last_peered': '2024-09-15T03:09:31.867950+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:09:31.867950+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5917, 'log_dups_size': 0, 'ondisk_log_size': 5917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48420466, 'num_objects': 5911, 'num_object_clones': 0, 'num_object_copies': 23644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5915, 'num_write_kb': 47300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3859}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5990", 'reported_seq': 6053, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:31.894575+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:09:31.894575+0000', 'last_peered': '2024-09-15T03:09:31.894575+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:09:31.894575+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5990, 'log_dups_size': 0, 'ondisk_log_size': 5990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49026674, 'num_objects': 5985, 'num_object_clones': 0, 'num_object_copies': 23940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5988, 'num_write_kb': 47892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3904}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5918", 'reported_seq': 5975, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.211627+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:09:32.211627+0000', 'last_peered': '2024-09-15T03:09:32.211627+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:09:32.211627+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5918, 'log_dups_size': 0, 'ondisk_log_size': 5918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48409828, 'num_objects': 5910, 'num_object_clones': 0, 'num_object_copies': 23640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5915, 'num_write_kb': 47297, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3868}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:37.040 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5928", 'reported_seq': 6011, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.627096+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:09:32.627096+0000', 'last_peered': '2024-09-15T03:09:32.627096+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:09:32.627096+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5928, 'log_dups_size': 0, 'ondisk_log_size': 5928, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48543346, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5926, 'num_write_kb': 47406, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3868}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5882", 'reported_seq': 5940, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.273367+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:09:32.273367+0000', 'last_peered': '2024-09-15T03:09:32.273367+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:09:32.273367+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5882, 'log_dups_size': 0, 'ondisk_log_size': 5882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48140715, 'num_objects': 5877, 'num_object_clones': 0, 'num_object_copies': 23508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5879, 'num_write_kb': 47020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3799}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5888", 'reported_seq': 9630, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.261027+0000', 'last_change': '2024-09-15T03:06:09.151203+0000', 'last_active': '2024-09-15T03:09:32.261027+0000', 'last_peered': '2024-09-15T03:09:32.261027+0000', 'last_clean': '2024-09-15T03:04:56.843143+0000', 'last_became_active': '2024-09-15T03:06:09.123678+0000', 'last_became_peered': '2024-09-15T03:06:09.123678+0000', 'last_unstale': '2024-09-15T03:09:32.261027+0000', 'last_undegraded': '2024-09-15T03:06:08.918189+0000', 'last_fullsized': '2024-09-15T03:06:08.838228+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5888, 'log_dups_size': 0, 'ondisk_log_size': 5888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48225081, 'num_objects': 5887, 'num_object_clones': 0, 'num_object_copies': 23548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 141, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5887, 'num_write_kb': 47095, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1853, 'num_bytes_recovered': 15163392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 5746}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'6004", 'reported_seq': 6081, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.707829+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:09:32.707829+0000', 'last_peered': '2024-09-15T03:09:32.707829+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:09:32.707829+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6004, 'log_dups_size': 0, 'ondisk_log_size': 6004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49175353, 'num_objects': 6003, 'num_object_clones': 0, 'num_object_copies': 24012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6003, 'num_write_kb': 48023, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3964}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5780", 'reported_seq': 5830, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.127615+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:09:32.127615+0000', 'last_peered': '2024-09-15T03:09:32.127615+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:09:32.127615+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5780, 'log_dups_size': 0, 'ondisk_log_size': 5780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47314546, 'num_objects': 5776, 'num_object_clones': 0, 'num_object_copies': 23104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5778, 'num_write_kb': 46220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3740}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:37.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:09:37.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5895", 'reported_seq': 5976, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:32.712813+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:09:32.712813+0000', 'last_peered': '2024-09-15T03:09:32.712813+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:09:32.712813+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5895, 'log_dups_size': 0, 'ondisk_log_size': 5895, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48264818, 'num_objects': 5892, 'num_object_clones': 0, 'num_object_copies': 23568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5893, 'num_write_kb': 47141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3904}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:37.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:09:37.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5932", 'reported_seq': 5995, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:31.805096+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:09:31.805096+0000', 'last_peered': '2024-09-15T03:09:31.805096+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:09:31.805096+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5932, 'log_dups_size': 0, 'ondisk_log_size': 5932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48551538, 'num_objects': 5927, 'num_object_clones': 0, 'num_object_copies': 23708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5930, 'num_write_kb': 47428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3845}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:37.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:09:38.531 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:38.532 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30805/377376 objects degraded (8.163%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":301,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94344,"data_bytes":772831804,"bytes_used":2663387136,"bytes_avail":383883669504,"bytes_total":386547056640,"degraded_objects":30805,"degraded_total":377376,"degraded_ratio":0.081629462392944968,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":111201,"recovering_keys_per_sec":0,"num_objects_recovered":163,"num_bytes_recovered":1335296,"num_keys_recovered":0,"write_bytes_sec":1687804,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:09:38.639 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:09:38.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:40.309 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:40.317 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:40.345 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-15T03:09:38.746718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":779188820,"num_objects":95121,"num_object_clones":0,"num_object_copies":380484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95586,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":15720448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95613,"ondisk_log_size":95613,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2618584,"kb_used_data":1402716,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":374868776,"statfs":{"total":386547056640,"available":383865626624,"internally_reserved":0,"allocated":1436381184,"data_stored":1435748418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,1,5,7,6,6,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26624024,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3689,"num_write_kb":27753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":1892352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003065"},"pg_stats":[{"pgid":"1.f","version":"31'5969","reported_seq":6011,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.197548+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:37.197548+0000","last_peered":"2024-09-15T03:09:37.197548+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:37.197548+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5969,"log_dups_size":0,"ondisk_log_size":5969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48668672,"num_objects":5941,"num_object_clones":0,"num_object_copies":23764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5721","reported_seq":5783,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:36.807601+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:36.807601+0000","last_peered":"2024-09-15T03:09:36.807601+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:36.807601+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732914,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3707}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'6131","reported_seq":6213,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.695459+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:37.695459+0000","last_peered":"2024-09-15T03:09:37.695459+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:37.695459+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6131,"log_dups_size":0,"ondisk_log_size":6131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49879889,"num_objects":6090,"num_object_clones":0,"num_object_copies":24360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6130,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3988}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5827","reported_seq":5890,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:36.879407+0000","last_change":"2024-09-15T03:06:09.201047+0000","last_active":"2024-09-15T03:09:36.879407+0000","last_peered":"2024-09-15T03:09:36.879407+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:36.879407+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5827,"log_dups_size":0,"ondisk_log_size":5827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47519346,"num_objects":5801,"num_object_clones":0,"num_object_copies":23204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5825,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'6055","reported_seq":6111,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.265846+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:37.265846+0000","last_peered":"2024-09-15T03:09:37.265846+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:37.265846+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6055,"log_dups_size":0,"ondisk_log_size":6055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49362546,"num_objects":6026,"num_object_clones":0,"num_object_copies":24104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6053,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3919}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'6012","reported_seq":6056,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.314697+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:37.314697+0000","last_peered":"2024-09-15T03:09:37.314697+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:37.314697+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6012,"log_dups_size":0,"ondisk_log_size":6012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49037312,"num_objects":5986,"num_object_clones":0,"num_object_copies":23944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6012,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5993","reported_seq":6058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:36.985501+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:36.985501+0000","last_peered":"2024-09-15T03:09:36.985501+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:36.985501+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5993,"log_dups_size":0,"ondisk_log_size":5993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48862834,"num_objects":5965,"num_object_clones":0,"num_object_copies":23860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5991,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3913}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'6055","reported_seq":6118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:36.945434+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:36.945434+0000","last_peered":"2024-09-15T03:09:36.945434+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:36.945434+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6055,"log_dups_size":0,"ondisk_log_size":6055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49395314,"num_objects":6030,"num_object_clones":0,"num_object_copies":24120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6053,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5998","reported_seq":6055,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.180728+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:37.180728+0000","last_peered":"2024-09-15T03:09:37.180728+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:37.180728+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5998,"log_dups_size":0,"ondisk_log_size":5998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48917732,"num_objects":5972,"num_object_clones":0,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3930}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'6007","reported_seq":6090,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.542170+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:37.542170+0000","last_peered":"2024-09-15T03:09:37.542170+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:37.542170+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6007,"log_dups_size":0,"ondisk_log_size":6007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48830066,"num_objects":5961,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5960","reported_seq":6018,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.199634+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:37.199634+0000","last_peered":"2024-09-15T03:09:37.199634+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:37.199634+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5960,"log_dups_size":0,"ondisk_log_size":5960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48517547,"num_objects":5923,"num_object_clones":0,"num_object_copies":23692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3845}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5968","reported_seq":9842,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.323906+0000","last_change":"2024-09-15T03:06:09.151203+0000","last_active":"2024-09-15T03:09:37.323906+0000","last_peered":"2024-09-15T03:09:37.323906+0000","last_clean":"2024-09-15T03:04:56.843143+0000","last_became_active":"2024-09-15T03:06:09.123678+0000","last_became_peered":"2024-09-15T03:06:09.123678+0000","last_unstale":"2024-09-15T03:09:37.323906+0000","last_undegraded":"2024-09-15T03:06:08.918189+0000","last_fullsized":"2024-09-15T03:06:08.838228+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5968,"log_dups_size":0,"ondisk_log_size":5968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48585529,"num_objects":5931,"num_object_clones":0,"num_object_copies":23724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":73,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1921,"num_bytes_recovered":15720448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":5858}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'6087","reported_seq":6164,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.709233+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:37.709233+0000","last_peered":"2024-09-15T03:09:37.709233+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:37.709233+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6087,"log_dups_size":0,"ondisk_log_size":6087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49527609,"num_objects":6046,"num_object_clones":0,"num_object_copies":24184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6086,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5853","reported_seq":5903,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.171923+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:37.171923+0000","last_peered":"2024-09-15T03:09:37.171923+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:37.171923+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47748722,"num_objects":5829,"num_object_clones":0,"num_object_copies":23316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5851,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5962","reported_seq":6043,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:37.744952+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:37.744952+0000","last_peered":"2024-09-15T03:09:37.744952+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:37.744952+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5962,"log_dups_size":0,"ondisk_log_size":5962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48502386,"num_objects":5921,"num_object_clones":0,"num_object_copies":23684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5960,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3933}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'6015","reported_seq":6078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:36.792737+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:36.792737+0000","last_peered":"2024-09-15T03:09:36.792737+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:36.792737+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6015,"log_dups_size":0,"ondisk_log_size":6015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49100402,"num_objects":5994,"num_object_clones":0,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6013,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":779188820,"num_objects":95121,"num_object_clones":0,"num_object_copies":380484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95586,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":15720448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1433202688,"data_stored":1433202688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95613,"ondisk_log_size":95613,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691852,"kb_used_data":381508,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93679988,"statfs":{"total":96636764160,"available":95928307712,"internally_reserved":0,"allocated":390664192,"data_stored":390505214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":3.6619999999999999},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.82799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.51600000000000001},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":3.8999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":3.859}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682704,"kb_used_data":381192,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93689136,"statfs":{"total":96636764160,"available":95937675264,"internally_reserved":0,"allocated":390340608,"data_stored":390184776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":7.1260000000000003},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":3.202}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.376}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":3.2850000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560716,"kb_used_data":258692,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93811124,"statfs":{"total":96636764160,"available":96062590976,"internally_reserved":0,"allocated":264900608,"data_stored":264741630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.24299999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.375},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":1.7350000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":1.0549999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":683312,"kb_used_data":381324,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93688528,"statfs":{"total":96636764160,"available":95937052672,"internally_reserved":0,"allocated":390475776,"data_stored":390316798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,0,0,4,1,4,0,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":1.1739999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":1.139},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.25600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389885952,"data_stored":389885952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264667136,"data_stored":264667136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":389115904,"data_stored":389115904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":389533696,"data_stored":389533696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:43.347 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:44.854 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:44.865 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:44.885 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-15T03:09:44.756221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":769104468,"num_objects":93890,"num_object_clones":0,"num_object_copies":375560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96817,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":16564224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96844,"ondisk_log_size":96844,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2612232,"kb_used_data":1388052,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":374875128,"statfs":{"total":386547056640,"available":383872131072,"internally_reserved":0,"allocated":1421365248,"data_stored":1420711996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,11,8,12,8,11,8,2,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3727336,"num_objects":-454,"num_object_clones":0,"num_object_copies":-1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2446,"num_write_kb":7961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.009981"},"pg_stats":[{"pgid":"1.f","version":"33'6039","reported_seq":6083,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.375730+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:42.375730+0000","last_peered":"2024-09-15T03:09:42.375730+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:42.375730+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6039,"log_dups_size":0,"ondisk_log_size":6039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48095232,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'5797","reported_seq":5861,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.878398+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:41.878398+0000","last_peered":"2024-09-15T03:09:41.878398+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:41.878398+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5797,"log_dups_size":0,"ondisk_log_size":5797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46110322,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6198","reported_seq":6282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.737466+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:42.737466+0000","last_peered":"2024-09-15T03:09:42.737466+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:42.737466+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49331025,"num_objects":6023,"num_object_clones":0,"num_object_copies":24092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6023,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6197,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3921}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'5891","reported_seq":6017,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.992581+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:09:41.992581+0000","last_peered":"2024-09-15T03:09:41.992581+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:41.992581+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5891,"log_dups_size":0,"ondisk_log_size":5891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46995058,"num_objects":5737,"num_object_clones":0,"num_object_copies":22948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5889,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":245760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6139","reported_seq":6197,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.379513+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:42.379513+0000","last_peered":"2024-09-15T03:09:42.379513+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:42.379513+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48674418,"num_objects":5942,"num_object_clones":0,"num_object_copies":23768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6101","reported_seq":6147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.176131+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:42.176131+0000","last_peered":"2024-09-15T03:09:42.176131+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:42.176131+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6101,"log_dups_size":0,"ondisk_log_size":6101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48308224,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6101,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6085","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.972434+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:41.972434+0000","last_peered":"2024-09-15T03:09:41.972434+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:41.972434+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6085,"log_dups_size":0,"ondisk_log_size":6085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48109170,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3821}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'6125","reported_seq":6190,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.979822+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:41.979822+0000","last_peered":"2024-09-15T03:09:41.979822+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:41.979822+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821874,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6069","reported_seq":6128,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.393700+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:42.393700+0000","last_peered":"2024-09-15T03:09:42.393700+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:42.393700+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6069,"log_dups_size":0,"ondisk_log_size":6069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48336100,"num_objects":5901,"num_object_clones":0,"num_object_copies":23604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6066,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3859}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6085","reported_seq":6170,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.443289+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:42.443289+0000","last_peered":"2024-09-15T03:09:42.443289+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:42.443289+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6085,"log_dups_size":0,"ondisk_log_size":6085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48191090,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3825}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6038","reported_seq":6098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.199419+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:42.199419+0000","last_peered":"2024-09-15T03:09:42.199419+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:42.199419+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47878571,"num_objects":5845,"num_object_clones":0,"num_object_copies":23380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6035,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6045","reported_seq":10084,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:09:42.784134+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:09:42.784134+0000","last_peered":"2024-09-15T03:09:42.784134+0000","last_clean":"2024-09-15T03:09:42.784134+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:09:42.784134+0000","last_undegraded":"2024-09-15T03:09:42.784134+0000","last_fullsized":"2024-09-15T03:09:42.784134+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6045,"log_dups_size":0,"ondisk_log_size":6045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47954745,"num_objects":5854,"num_object_clones":0,"num_object_copies":23416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6044,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6187","reported_seq":6266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.740188+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:42.740188+0000","last_peered":"2024-09-15T03:09:42.740188+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:42.740188+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6187,"log_dups_size":0,"ondisk_log_size":6187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48708409,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5928","reported_seq":5979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.228331+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:42.228331+0000","last_peered":"2024-09-15T03:09:42.228331+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:42.228331+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47134322,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3718}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6024","reported_seq":6107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.563082+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:42.563082+0000","last_peered":"2024-09-15T03:09:42.563082+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:42.563082+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47994482,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3871}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6093","reported_seq":6158,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.987649+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:41.987649+0000","last_peered":"2024-09-15T03:09:41.987649+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:41.987649+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48461426,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":769104468,"num_objects":93890,"num_object_clones":0,"num_object_copies":375560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96817,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":16564224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418076160,"data_stored":1418076160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96844,"ondisk_log_size":96844,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674488,"kb_used_data":376616,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93697352,"statfs":{"total":96636764160,"available":95946088448,"internally_reserved":0,"allocated":385654784,"data_stored":385491713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,4,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":2.7690000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.49299999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.52100000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686028,"kb_used_data":376192,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93685812,"statfs":{"total":96636764160,"available":95934271488,"internally_reserved":0,"allocated":385220608,"data_stored":385060683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,2,5,7,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.215}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.25900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.65300000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569116,"kb_used_data":258768,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93802724,"statfs":{"total":96636764160,"available":96053989376,"internally_reserved":0,"allocated":264978432,"data_stored":264811247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,0,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.629},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.76000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.59899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.47299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682600,"kb_used_data":376476,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93689240,"statfs":{"total":96636764160,"available":95937781760,"internally_reserved":0,"allocated":385511424,"data_stored":385348353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,6,1,3,2,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.224}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.253},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":3.141}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":3.1099999999999999},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":3.1720000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384892928,"data_stored":384892928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264675328,"data_stored":264675328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":384073728,"data_stored":384073728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":384434176,"data_stored":384434176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:44.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:46.366 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:46.393 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-15T03:09:44.756221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":769104468,"num_objects":93890,"num_object_clones":0,"num_object_copies":375560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96817,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":16564224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96844,"ondisk_log_size":96844,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2612232,"kb_used_data":1388052,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":374875128,"statfs":{"total":386547056640,"available":383872131072,"internally_reserved":0,"allocated":1421365248,"data_stored":1420711996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,11,8,12,8,11,8,2,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3727336,"num_objects":-454,"num_object_clones":0,"num_object_copies":-1816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2446,"num_write_kb":7961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.009981"},"pg_stats":[{"pgid":"1.f","version":"33'6039","reported_seq":6083,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.375730+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:42.375730+0000","last_peered":"2024-09-15T03:09:42.375730+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:42.375730+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6039,"log_dups_size":0,"ondisk_log_size":6039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48095232,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6039,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"32'5797","reported_seq":5861,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.878398+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:41.878398+0000","last_peered":"2024-09-15T03:09:41.878398+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:41.878398+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5797,"log_dups_size":0,"ondisk_log_size":5797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46110322,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6198","reported_seq":6282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.737466+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:42.737466+0000","last_peered":"2024-09-15T03:09:42.737466+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:42.737466+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49331025,"num_objects":6023,"num_object_clones":0,"num_object_copies":24092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6023,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6197,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3921}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"32'5891","reported_seq":6017,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.992581+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:09:41.992581+0000","last_peered":"2024-09-15T03:09:41.992581+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:41.992581+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5891,"log_dups_size":0,"ondisk_log_size":5891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46995058,"num_objects":5737,"num_object_clones":0,"num_object_copies":22948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5889,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":245760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3824}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6139","reported_seq":6197,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.379513+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:42.379513+0000","last_peered":"2024-09-15T03:09:42.379513+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:42.379513+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48674418,"num_objects":5942,"num_object_clones":0,"num_object_copies":23768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6137,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6101","reported_seq":6147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.176131+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:42.176131+0000","last_peered":"2024-09-15T03:09:42.176131+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:42.176131+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6101,"log_dups_size":0,"ondisk_log_size":6101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48308224,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6101,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6085","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.972434+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:41.972434+0000","last_peered":"2024-09-15T03:09:41.972434+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:41.972434+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6085,"log_dups_size":0,"ondisk_log_size":6085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48109170,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3821}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"32'6125","reported_seq":6190,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.979822+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:41.979822+0000","last_peered":"2024-09-15T03:09:41.979822+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:41.979822+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821874,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6123,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6069","reported_seq":6128,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.393700+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:42.393700+0000","last_peered":"2024-09-15T03:09:42.393700+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:42.393700+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6069,"log_dups_size":0,"ondisk_log_size":6069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48336100,"num_objects":5901,"num_object_clones":0,"num_object_copies":23604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6066,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3859}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6085","reported_seq":6170,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.443289+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:42.443289+0000","last_peered":"2024-09-15T03:09:42.443289+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:42.443289+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6085,"log_dups_size":0,"ondisk_log_size":6085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48191090,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3825}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6038","reported_seq":6098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.199419+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:42.199419+0000","last_peered":"2024-09-15T03:09:42.199419+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:42.199419+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47878571,"num_objects":5845,"num_object_clones":0,"num_object_copies":23380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6035,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3767}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6045","reported_seq":10084,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:09:42.784134+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:09:42.784134+0000","last_peered":"2024-09-15T03:09:42.784134+0000","last_clean":"2024-09-15T03:09:42.784134+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:09:42.784134+0000","last_undegraded":"2024-09-15T03:09:42.784134+0000","last_fullsized":"2024-09-15T03:09:42.784134+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6045,"log_dups_size":0,"ondisk_log_size":6045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47954745,"num_objects":5854,"num_object_clones":0,"num_object_copies":23416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6044,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6187","reported_seq":6266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.740188+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:42.740188+0000","last_peered":"2024-09-15T03:09:42.740188+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:42.740188+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6187,"log_dups_size":0,"ondisk_log_size":6187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48708409,"num_objects":5946,"num_object_clones":0,"num_object_copies":23784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6186,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5928","reported_seq":5979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.228331+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:42.228331+0000","last_peered":"2024-09-15T03:09:42.228331+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:42.228331+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5928,"log_dups_size":0,"ondisk_log_size":5928,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47134322,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5926,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3718}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6024","reported_seq":6107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:42.563082+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:42.563082+0000","last_peered":"2024-09-15T03:09:42.563082+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:42.563082+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6024,"log_dups_size":0,"ondisk_log_size":6024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47994482,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6022,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3871}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6093","reported_seq":6158,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:41.987649+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:41.987649+0000","last_peered":"2024-09-15T03:09:41.987649+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:41.987649+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48461426,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":769104468,"num_objects":93890,"num_object_clones":0,"num_object_copies":375560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96817,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":16564224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1418076160,"data_stored":1418076160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96844,"ondisk_log_size":96844,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674488,"kb_used_data":376616,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93697352,"statfs":{"total":96636764160,"available":95946088448,"internally_reserved":0,"allocated":385654784,"data_stored":385491713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,4,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":2.7690000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.49299999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.52100000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686028,"kb_used_data":376192,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93685812,"statfs":{"total":96636764160,"available":95934271488,"internally_reserved":0,"allocated":385220608,"data_stored":385060683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,2,5,7,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.215}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.25900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.65300000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569116,"kb_used_data":258768,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93802724,"statfs":{"total":96636764160,"available":96053989376,"internally_reserved":0,"allocated":264978432,"data_stored":264811247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,0,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.629},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.76000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.59899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.47299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682600,"kb_used_data":376476,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93689240,"statfs":{"total":96636764160,"available":95937781760,"internally_reserved":0,"allocated":385511424,"data_stored":385348353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,6,1,3,2,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.224}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.253},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":3.141}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":3.1099999999999999},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":3.1720000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384892928,"data_stored":384892928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264675328,"data_stored":264675328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":384073728,"data_stored":384073728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":384434176,"data_stored":384434176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:46.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:09:46.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6039", 'reported_seq': 6083, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.375730+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:09:42.375730+0000', 'last_peered': '2024-09-15T03:09:42.375730+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:09:42.375730+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6039, 'log_dups_size': 0, 'ondisk_log_size': 6039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48095232, 'num_objects': 5871, 'num_object_clones': 0, 'num_object_copies': 23484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6039, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3895}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:46.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:09:46.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "32'5797", 'reported_seq': 5861, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:41.878398+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:09:41.878398+0000', 'last_peered': '2024-09-15T03:09:41.878398+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:09:41.878398+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5797, 'log_dups_size': 0, 'ondisk_log_size': 5797, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46110322, 'num_objects': 5629, 'num_object_clones': 0, 'num_object_copies': 22516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2000, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5795, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6198", 'reported_seq': 6282, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.737466+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:09:42.737466+0000', 'last_peered': '2024-09-15T03:09:42.737466+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:09:42.737466+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6198, 'log_dups_size': 0, 'ondisk_log_size': 6198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49331025, 'num_objects': 6023, 'num_object_clones': 0, 'num_object_copies': 24092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6023, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6197, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3921}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "32'5891", 'reported_seq': 6017, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:41.992581+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:09:41.992581+0000', 'last_peered': '2024-09-15T03:09:41.992581+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:09:41.992581+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5891, 'log_dups_size': 0, 'ondisk_log_size': 5891, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46995058, 'num_objects': 5737, 'num_object_clones': 0, 'num_object_copies': 22948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1913, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5889, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 245760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6139", 'reported_seq': 6197, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.379513+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:09:42.379513+0000', 'last_peered': '2024-09-15T03:09:42.379513+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:09:42.379513+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6139, 'log_dups_size': 0, 'ondisk_log_size': 6139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48674418, 'num_objects': 5942, 'num_object_clones': 0, 'num_object_copies': 23768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6137, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3835}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6101", 'reported_seq': 6147, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.176131+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:09:42.176131+0000', 'last_peered': '2024-09-15T03:09:42.176131+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:09:42.176131+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6101, 'log_dups_size': 0, 'ondisk_log_size': 6101, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48308224, 'num_objects': 5897, 'num_object_clones': 0, 'num_object_copies': 23588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6101, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3841}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:09:46.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6085", 'reported_seq': 6152, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:41.972434+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:09:41.972434+0000', 'last_peered': '2024-09-15T03:09:41.972434+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:09:41.972434+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6085, 'log_dups_size': 0, 'ondisk_log_size': 6085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48109170, 'num_objects': 5873, 'num_object_clones': 0, 'num_object_copies': 23492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6083, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3821}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "32'6125", 'reported_seq': 6190, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:41.979822+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:09:41.979822+0000', 'last_peered': '2024-09-15T03:09:41.979822+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:09:41.979822+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6125, 'log_dups_size': 0, 'ondisk_log_size': 6125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48821874, 'num_objects': 5960, 'num_object_clones': 0, 'num_object_copies': 23840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6123, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6069", 'reported_seq': 6128, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.393700+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:09:42.393700+0000', 'last_peered': '2024-09-15T03:09:42.393700+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:09:42.393700+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6069, 'log_dups_size': 0, 'ondisk_log_size': 6069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48336100, 'num_objects': 5901, 'num_object_clones': 0, 'num_object_copies': 23604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6066, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3859}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6085", 'reported_seq': 6170, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.443289+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:09:42.443289+0000', 'last_peered': '2024-09-15T03:09:42.443289+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:09:42.443289+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6085, 'log_dups_size': 0, 'ondisk_log_size': 6085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48191090, 'num_objects': 5883, 'num_object_clones': 0, 'num_object_copies': 23532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6083, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3825}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6038", 'reported_seq': 6098, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.199419+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:09:42.199419+0000', 'last_peered': '2024-09-15T03:09:42.199419+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:09:42.199419+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6038, 'log_dups_size': 0, 'ondisk_log_size': 6038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47878571, 'num_objects': 5845, 'num_object_clones': 0, 'num_object_copies': 23380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6035, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3767}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:09:46.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6187", 'reported_seq': 6266, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.740188+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:09:42.740188+0000', 'last_peered': '2024-09-15T03:09:42.740188+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:09:42.740188+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6187, 'log_dups_size': 0, 'ondisk_log_size': 6187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48708409, 'num_objects': 5946, 'num_object_clones': 0, 'num_object_copies': 23784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6186, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:46.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:09:46.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5928", 'reported_seq': 5979, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.228331+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:09:42.228331+0000', 'last_peered': '2024-09-15T03:09:42.228331+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:09:42.228331+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5928, 'log_dups_size': 0, 'ondisk_log_size': 5928, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47134322, 'num_objects': 5754, 'num_object_clones': 0, 'num_object_copies': 23016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5926, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3718}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:46.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:09:46.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6024", 'reported_seq': 6107, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:42.563082+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:09:42.563082+0000', 'last_peered': '2024-09-15T03:09:42.563082+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:09:42.563082+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6024, 'log_dups_size': 0, 'ondisk_log_size': 6024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47994482, 'num_objects': 5859, 'num_object_clones': 0, 'num_object_copies': 23436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6022, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:46.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:09:46.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6093", 'reported_seq': 6158, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:41.987649+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:09:41.987649+0000', 'last_peered': '2024-09-15T03:09:41.987649+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:09:41.987649+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6093, 'log_dups_size': 0, 'ondisk_log_size': 6093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48461426, 'num_objects': 5916, 'num_object_clones': 0, 'num_object_copies': 23664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6091, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3834}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:46.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:09:48.139 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:48.140 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30632/375560 objects degraded (8.156%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":311,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93890,"data_bytes":769104468,"bytes_used":2674925568,"bytes_avail":383872131072,"bytes_total":386547056640,"degraded_objects":30632,"degraded_total":375560,"degraded_ratio":0.081563531792523167,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":105340,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":843776,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":153},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:09:48.252 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:09:48.252 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:49.479 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:49.505 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:49.513 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-15T03:09:48.757162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":758839892,"num_objects":92637,"num_object_clones":0,"num_object_copies":370548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92637,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98070,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":16883712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98097,"ondisk_log_size":98097,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2583784,"kb_used_data":1372104,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":374903576,"statfs":{"total":386547056640,"available":383901261824,"internally_reserved":0,"allocated":1405034496,"data_stored":1404368902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,8,6,14,9,10,2,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20348928,"num_objects":-2484,"num_object_clones":0,"num_object_copies":-9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2484,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1163264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006675"},"pg_stats":[{"pgid":"1.f","version":"33'6119","reported_seq":6163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.391707+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:47.391707+0000","last_peered":"2024-09-15T03:09:47.391707+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:47.391707+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47439872,"num_objects":5791,"num_object_clones":0,"num_object_copies":23164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6119,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5871","reported_seq":5939,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.034667+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:47.034667+0000","last_peered":"2024-09-15T03:09:47.034667+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:47.034667+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5871,"log_dups_size":0,"ondisk_log_size":5871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45504114,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5869,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6274","reported_seq":6358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.764747+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:47.764747+0000","last_peered":"2024-09-15T03:09:47.764747+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:47.764747+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48708433,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6273,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3845}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5979","reported_seq":6183,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:46.950514+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:09:46.950514+0000","last_peered":"2024-09-15T03:09:46.950514+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:46.950514+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5979,"log_dups_size":0,"ondisk_log_size":5979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46274162,"num_objects":5649,"num_object_clones":0,"num_object_copies":22596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5977,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":565248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3775}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6221","reported_seq":6278,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.421713+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:47.421713+0000","last_peered":"2024-09-15T03:09:47.421713+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:47.421713+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6221,"log_dups_size":0,"ondisk_log_size":6221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48002674,"num_objects":5860,"num_object_clones":0,"num_object_copies":23440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6181","reported_seq":6227,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.387932+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:47.387932+0000","last_peered":"2024-09-15T03:09:47.387932+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:47.387932+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6181,"log_dups_size":0,"ondisk_log_size":6181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47652864,"num_objects":5817,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6181,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6162","reported_seq":6231,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.022369+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:47.022369+0000","last_peered":"2024-09-15T03:09:47.022369+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:47.022369+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6162,"log_dups_size":0,"ondisk_log_size":6162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47478386,"num_objects":5796,"num_object_clones":0,"num_object_copies":23184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6160,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3745}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6199","reported_seq":6266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.075195+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:47.075195+0000","last_peered":"2024-09-15T03:09:47.075195+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:47.075195+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48215666,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6151","reported_seq":6210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.343587+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:47.343587+0000","last_peered":"2024-09-15T03:09:47.343587+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:47.343587+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6151,"log_dups_size":0,"ondisk_log_size":6151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47664356,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6165","reported_seq":6250,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.729661+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:47.729661+0000","last_peered":"2024-09-15T03:09:47.729661+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:47.729661+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6165,"log_dups_size":0,"ondisk_log_size":6165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47535730,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6123","reported_seq":6185,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.442686+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:47.442686+0000","last_peered":"2024-09-15T03:09:47.442686+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:47.442686+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6123,"log_dups_size":0,"ondisk_log_size":6123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47182251,"num_objects":5760,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3683}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6122","reported_seq":10160,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:09:47.908840+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:09:47.908840+0000","last_peered":"2024-09-15T03:09:47.908840+0000","last_clean":"2024-09-15T03:09:47.908840+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:09:47.908840+0000","last_undegraded":"2024-09-15T03:09:47.908840+0000","last_fullsized":"2024-09-15T03:09:47.908840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6122,"log_dups_size":0,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323961,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6260","reported_seq":6339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.615892+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:47.615892+0000","last_peered":"2024-09-15T03:09:47.615892+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:47.615892+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48110393,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6259,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6002","reported_seq":6054,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.308583+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:47.308583+0000","last_peered":"2024-09-15T03:09:47.308583+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:47.308583+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6002,"log_dups_size":0,"ondisk_log_size":6002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46528114,"num_objects":5680,"num_object_clones":0,"num_object_copies":22720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6000,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3644}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6103","reported_seq":6186,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.770246+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:47.770246+0000","last_peered":"2024-09-15T03:09:47.770246+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:47.770246+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6103,"log_dups_size":0,"ondisk_log_size":6103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47347314,"num_objects":5780,"num_object_clones":0,"num_object_copies":23120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6101,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3792}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6165","reported_seq":6230,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:46.930377+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:46.930377+0000","last_peered":"2024-09-15T03:09:46.930377+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:46.930377+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6165,"log_dups_size":0,"ondisk_log_size":6165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47871602,"num_objects":5844,"num_object_clones":0,"num_object_copies":23376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":758839892,"num_objects":92637,"num_object_clones":0,"num_object_copies":370548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92637,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98070,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":16883712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1402388480,"data_stored":1402388480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98097,"ondisk_log_size":98097,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677036,"kb_used_data":370852,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93694804,"statfs":{"total":96636764160,"available":95943479296,"internally_reserved":0,"allocated":379752448,"data_stored":379585263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.215},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.30399999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":3.992},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.246}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":4.0510000000000002},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.41099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668644,"kb_used_data":371292,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93703196,"statfs":{"total":96636764160,"available":95952072704,"internally_reserved":0,"allocated":380203008,"data_stored":380038969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":19.329000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.308},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":22.710000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556324,"kb_used_data":258460,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93815516,"statfs":{"total":96636764160,"available":96067088384,"internally_reserved":0,"allocated":264663040,"data_stored":264495855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":1.198},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":1.2769999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.32600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.51400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":681780,"kb_used_data":371500,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93690060,"statfs":{"total":96636764160,"available":95938621440,"internally_reserved":0,"allocated":380416000,"data_stored":380248815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,5,3,1,2,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.1789999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.30299999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":3.222},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":2.2200000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.26100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379719680,"data_stored":379719680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264364032,"data_stored":264364032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":378953728,"data_stored":378953728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379351040,"data_stored":379351040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:52.514 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:53.501 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:53.516 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:53.553 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-15T03:09:52.760088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751901268,"num_objects":91790,"num_object_clones":0,"num_object_copies":367160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98917,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2158,"num_bytes_recovered":17326080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98944,"ondisk_log_size":98944,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2573948,"kb_used_data":1362260,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":374913412,"statfs":{"total":386547056640,"available":383911333888,"internally_reserved":0,"allocated":1394954240,"data_stored":1394288646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,12,5,10,8,11,3,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27287552,"num_objects":-3331,"num_object_clones":0,"num_object_copies":-13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.805913"},"pg_stats":[{"pgid":"1.f","version":"33'6178","reported_seq":6222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.411766+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:52.411766+0000","last_peered":"2024-09-15T03:09:52.411766+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:52.411766+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6178,"log_dups_size":0,"ondisk_log_size":6178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46956544,"num_objects":5732,"num_object_clones":0,"num_object_copies":22928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6178,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5948","reported_seq":6016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:51.948459+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:51.948459+0000","last_peered":"2024-09-15T03:09:51.948459+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:51.948459+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44873330,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3482}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6274","reported_seq":6358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.764747+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:47.764747+0000","last_peered":"2024-09-15T03:09:47.764747+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:47.764747+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48708433,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6273,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3845}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6054","reported_seq":6378,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.126577+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:09:52.126577+0000","last_peered":"2024-09-15T03:09:52.126577+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:52.126577+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45659762,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6305","reported_seq":6363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.452922+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:52.452922+0000","last_peered":"2024-09-15T03:09:52.452922+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:52.452922+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6305,"log_dups_size":0,"ondisk_log_size":6305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314546,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6254","reported_seq":6300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.438858+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:52.438858+0000","last_peered":"2024-09-15T03:09:52.438858+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:52.438858+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47054848,"num_objects":5744,"num_object_clones":0,"num_object_copies":22976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6238","reported_seq":6311,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:51.776847+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:51.776847+0000","last_peered":"2024-09-15T03:09:51.776847+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:51.776847+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6238,"log_dups_size":0,"ondisk_log_size":6238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855794,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6236,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3671}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6280","reported_seq":6351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.038869+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:52.038869+0000","last_peered":"2024-09-15T03:09:52.038869+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:52.038869+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6280,"log_dups_size":0,"ondisk_log_size":6280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47552114,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6231","reported_seq":6290,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.266444+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:52.266444+0000","last_peered":"2024-09-15T03:09:52.266444+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:52.266444+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6231,"log_dups_size":0,"ondisk_log_size":6231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47008996,"num_objects":5739,"num_object_clones":0,"num_object_copies":22956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6165","reported_seq":6250,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.729661+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:47.729661+0000","last_peered":"2024-09-15T03:09:47.729661+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:47.729661+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6165,"log_dups_size":0,"ondisk_log_size":6165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47535730,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6199","reported_seq":6261,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.259897+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:52.259897+0000","last_peered":"2024-09-15T03:09:52.259897+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:52.259897+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46559659,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6122","reported_seq":10160,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:09:47.908840+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:09:47.908840+0000","last_peered":"2024-09-15T03:09:47.908840+0000","last_clean":"2024-09-15T03:09:47.908840+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:09:47.908840+0000","last_undegraded":"2024-09-15T03:09:47.908840+0000","last_fullsized":"2024-09-15T03:09:47.908840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6122,"log_dups_size":0,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323961,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6260","reported_seq":6339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.615892+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:47.615892+0000","last_peered":"2024-09-15T03:09:47.615892+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:47.615892+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48110393,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6259,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6080","reported_seq":6132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.322275+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:52.322275+0000","last_peered":"2024-09-15T03:09:52.322275+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:52.322275+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45889138,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6078,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3566}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6103","reported_seq":6186,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.770246+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:47.770246+0000","last_peered":"2024-09-15T03:09:47.770246+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:47.770246+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6103,"log_dups_size":0,"ondisk_log_size":6103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47347314,"num_objects":5780,"num_object_clones":0,"num_object_copies":23120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6101,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3792}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6253","reported_seq":6315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.132154+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:52.132154+0000","last_peered":"2024-09-15T03:09:52.132154+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:52.132154+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6253,"log_dups_size":0,"ondisk_log_size":6253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150706,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3674}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751901268,"num_objects":91790,"num_object_clones":0,"num_object_copies":367160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98917,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2158,"num_bytes_recovered":17326080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1392246784,"data_stored":1392246784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98944,"ondisk_log_size":98944,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680416,"kb_used_data":365904,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93691424,"statfs":{"total":96636764160,"available":95940018176,"internally_reserved":0,"allocated":374685696,"data_stored":374518511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,4,1,1,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.34699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.315},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.46700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668644,"kb_used_data":371292,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93703196,"statfs":{"total":96636764160,"available":95952072704,"internally_reserved":0,"allocated":380203008,"data_stored":380038969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":19.329000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.308},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":22.710000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556324,"kb_used_data":258460,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93815516,"statfs":{"total":96636764160,"available":96067088384,"internally_reserved":0,"allocated":264663040,"data_stored":264495855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":1.198},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":1.2769999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.32600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.51400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668564,"kb_used_data":366604,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93703276,"statfs":{"total":96636764160,"available":95952154624,"internally_reserved":0,"allocated":375402496,"data_stored":375235311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,2,2,1,3,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.222},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.29299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.375},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374644736,"data_stored":374644736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264364032,"data_stored":264364032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":378953728,"data_stored":378953728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374284288,"data_stored":374284288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:53.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:54.378 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:54.389 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:54.394 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-15T03:09:52.760088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":751901268,"num_objects":91790,"num_object_clones":0,"num_object_copies":367160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98917,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2158,"num_bytes_recovered":17326080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98944,"ondisk_log_size":98944,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2573948,"kb_used_data":1362260,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":374913412,"statfs":{"total":386547056640,"available":383911333888,"internally_reserved":0,"allocated":1394954240,"data_stored":1394288646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,12,5,10,8,11,3,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27287552,"num_objects":-3331,"num_object_clones":0,"num_object_copies":-13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1605632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.805913"},"pg_stats":[{"pgid":"1.f","version":"33'6178","reported_seq":6222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.411766+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:52.411766+0000","last_peered":"2024-09-15T03:09:52.411766+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:52.411766+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6178,"log_dups_size":0,"ondisk_log_size":6178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46956544,"num_objects":5732,"num_object_clones":0,"num_object_copies":22928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6178,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5948","reported_seq":6016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:51.948459+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:51.948459+0000","last_peered":"2024-09-15T03:09:51.948459+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:51.948459+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44873330,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3482}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6274","reported_seq":6358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.764747+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:47.764747+0000","last_peered":"2024-09-15T03:09:47.764747+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:47.764747+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48708433,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6273,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3845}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6054","reported_seq":6378,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.126577+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:09:52.126577+0000","last_peered":"2024-09-15T03:09:52.126577+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:52.126577+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45659762,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6305","reported_seq":6363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.452922+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:52.452922+0000","last_peered":"2024-09-15T03:09:52.452922+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:52.452922+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6305,"log_dups_size":0,"ondisk_log_size":6305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314546,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6254","reported_seq":6300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.438858+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:52.438858+0000","last_peered":"2024-09-15T03:09:52.438858+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:52.438858+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47054848,"num_objects":5744,"num_object_clones":0,"num_object_copies":22976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6238","reported_seq":6311,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:51.776847+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:51.776847+0000","last_peered":"2024-09-15T03:09:51.776847+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:51.776847+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6238,"log_dups_size":0,"ondisk_log_size":6238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855794,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6236,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3671}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6280","reported_seq":6351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.038869+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:52.038869+0000","last_peered":"2024-09-15T03:09:52.038869+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:52.038869+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6280,"log_dups_size":0,"ondisk_log_size":6280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47552114,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6231","reported_seq":6290,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.266444+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:52.266444+0000","last_peered":"2024-09-15T03:09:52.266444+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:52.266444+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6231,"log_dups_size":0,"ondisk_log_size":6231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47008996,"num_objects":5739,"num_object_clones":0,"num_object_copies":22956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6165","reported_seq":6250,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.729661+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:47.729661+0000","last_peered":"2024-09-15T03:09:47.729661+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:47.729661+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6165,"log_dups_size":0,"ondisk_log_size":6165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47535730,"num_objects":5803,"num_object_clones":0,"num_object_copies":23212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6163,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6199","reported_seq":6261,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.259897+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:52.259897+0000","last_peered":"2024-09-15T03:09:52.259897+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:52.259897+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46559659,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6122","reported_seq":10160,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:09:47.908840+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:09:47.908840+0000","last_peered":"2024-09-15T03:09:47.908840+0000","last_clean":"2024-09-15T03:09:47.908840+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:09:47.908840+0000","last_undegraded":"2024-09-15T03:09:47.908840+0000","last_fullsized":"2024-09-15T03:09:47.908840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6122,"log_dups_size":0,"ondisk_log_size":6122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47323961,"num_objects":5777,"num_object_clones":0,"num_object_copies":23108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6121,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6260","reported_seq":6339,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.615892+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:47.615892+0000","last_peered":"2024-09-15T03:09:47.615892+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:47.615892+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48110393,"num_objects":5873,"num_object_clones":0,"num_object_copies":23492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6259,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3834}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6080","reported_seq":6132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.322275+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:52.322275+0000","last_peered":"2024-09-15T03:09:52.322275+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:52.322275+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45889138,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6078,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3566}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6103","reported_seq":6186,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:47.770246+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:47.770246+0000","last_peered":"2024-09-15T03:09:47.770246+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:47.770246+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6103,"log_dups_size":0,"ondisk_log_size":6103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47347314,"num_objects":5780,"num_object_clones":0,"num_object_copies":23120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6101,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3792}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6253","reported_seq":6315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.132154+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:52.132154+0000","last_peered":"2024-09-15T03:09:52.132154+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:52.132154+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6253,"log_dups_size":0,"ondisk_log_size":6253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150706,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3674}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":751901268,"num_objects":91790,"num_object_clones":0,"num_object_copies":367160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91790,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98917,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2158,"num_bytes_recovered":17326080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1392246784,"data_stored":1392246784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98944,"ondisk_log_size":98944,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680416,"kb_used_data":365904,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93691424,"statfs":{"total":96636764160,"available":95940018176,"internally_reserved":0,"allocated":374685696,"data_stored":374518511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,4,1,1,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.34699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.315},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.46700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668644,"kb_used_data":371292,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93703196,"statfs":{"total":96636764160,"available":95952072704,"internally_reserved":0,"allocated":380203008,"data_stored":380038969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":19.329000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.308},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.35999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":22.710000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556324,"kb_used_data":258460,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93815516,"statfs":{"total":96636764160,"available":96067088384,"internally_reserved":0,"allocated":264663040,"data_stored":264495855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":1.198},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":1.2769999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.32600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.51400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668564,"kb_used_data":366604,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93703276,"statfs":{"total":96636764160,"available":95952154624,"internally_reserved":0,"allocated":375402496,"data_stored":375235311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,2,2,1,3,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.222},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.29299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.375},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374644736,"data_stored":374644736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264364032,"data_stored":264364032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":378953728,"data_stored":378953728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374284288,"data_stored":374284288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:09:54.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:09:54.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6178", 'reported_seq': 6222, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.411766+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:09:52.411766+0000', 'last_peered': '2024-09-15T03:09:52.411766+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:09:52.411766+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6178, 'log_dups_size': 0, 'ondisk_log_size': 6178, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46956544, 'num_objects': 5732, 'num_object_clones': 0, 'num_object_copies': 22928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6178, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3756}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:54.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:09:54.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5948", 'reported_seq': 6016, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:51.948459+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:09:51.948459+0000', 'last_peered': '2024-09-15T03:09:51.948459+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:09:51.948459+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 5948, 'log_dups_size': 0, 'ondisk_log_size': 5948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44873330, 'num_objects': 5478, 'num_object_clones': 0, 'num_object_copies': 21912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1998, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5946, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:54.395 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:09:54.395 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6274", 'reported_seq': 6358, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:47.764747+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:09:47.764747+0000', 'last_peered': '2024-09-15T03:09:47.764747+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:09:47.764747+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6274, 'log_dups_size': 0, 'ondisk_log_size': 6274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48708433, 'num_objects': 5947, 'num_object_clones': 0, 'num_object_copies': 23788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2103, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5947, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6273, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3845}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6054", 'reported_seq': 6378, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.126577+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:09:52.126577+0000', 'last_peered': '2024-09-15T03:09:52.126577+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:09:52.126577+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6054, 'log_dups_size': 0, 'ondisk_log_size': 6054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45659762, 'num_objects': 5574, 'num_object_clones': 0, 'num_object_copies': 22296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1814, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6052, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 131, 'num_bytes_recovered': 1007616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6305", 'reported_seq': 6363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.452922+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:09:52.452922+0000', 'last_peered': '2024-09-15T03:09:52.452922+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:09:52.452922+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6305, 'log_dups_size': 0, 'ondisk_log_size': 6305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47314546, 'num_objects': 5776, 'num_object_clones': 0, 'num_object_copies': 23104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6303, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3669}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6254", 'reported_seq': 6300, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.438858+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:09:52.438858+0000', 'last_peered': '2024-09-15T03:09:52.438858+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:09:52.438858+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6254, 'log_dups_size': 0, 'ondisk_log_size': 6254, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47054848, 'num_objects': 5744, 'num_object_clones': 0, 'num_object_copies': 22976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6254, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3688}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6238", 'reported_seq': 6311, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:51.776847+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:09:51.776847+0000', 'last_peered': '2024-09-15T03:09:51.776847+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:09:51.776847+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6238, 'log_dups_size': 0, 'ondisk_log_size': 6238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46855794, 'num_objects': 5720, 'num_object_clones': 0, 'num_object_copies': 22880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6236, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3671}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:09:54.396 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6280", 'reported_seq': 6351, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.038869+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:09:52.038869+0000', 'last_peered': '2024-09-15T03:09:52.038869+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:09:52.038869+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6280, 'log_dups_size': 0, 'ondisk_log_size': 6280, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47552114, 'num_objects': 5805, 'num_object_clones': 0, 'num_object_copies': 23220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2080, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6278, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3727}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6231", 'reported_seq': 6290, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.266444+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:09:52.266444+0000', 'last_peered': '2024-09-15T03:09:52.266444+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:09:52.266444+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6231, 'log_dups_size': 0, 'ondisk_log_size': 6231, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47008996, 'num_objects': 5739, 'num_object_clones': 0, 'num_object_copies': 22956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5739, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6228, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3697}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6165", 'reported_seq': 6250, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:47.729661+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:09:47.729661+0000', 'last_peered': '2024-09-15T03:09:47.729661+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:09:47.729661+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6165, 'log_dups_size': 0, 'ondisk_log_size': 6165, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47535730, 'num_objects': 5803, 'num_object_clones': 0, 'num_object_copies': 23212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6163, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3745}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6199", 'reported_seq': 6261, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.259897+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:09:52.259897+0000', 'last_peered': '2024-09-15T03:09:52.259897+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:09:52.259897+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6199, 'log_dups_size': 0, 'ondisk_log_size': 6199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46559659, 'num_objects': 5684, 'num_object_clones': 0, 'num_object_copies': 22736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2080, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6196, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3607}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6260", 'reported_seq': 6339, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:47.615892+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:09:47.615892+0000', 'last_peered': '2024-09-15T03:09:47.615892+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:09:47.615892+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6260, 'log_dups_size': 0, 'ondisk_log_size': 6260, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48110393, 'num_objects': 5873, 'num_object_clones': 0, 'num_object_copies': 23492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6259, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3834}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:09:54.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6080", 'reported_seq': 6132, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.322275+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:09:52.322275+0000', 'last_peered': '2024-09-15T03:09:52.322275+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:09:52.322275+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6080, 'log_dups_size': 0, 'ondisk_log_size': 6080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45889138, 'num_objects': 5602, 'num_object_clones': 0, 'num_object_copies': 22408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6078, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3566}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:09:54.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:09:54.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6103", 'reported_seq': 6186, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:47.770246+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:09:47.770246+0000', 'last_peered': '2024-09-15T03:09:47.770246+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:09:47.770246+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6103, 'log_dups_size': 0, 'ondisk_log_size': 6103, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47347314, 'num_objects': 5780, 'num_object_clones': 0, 'num_object_copies': 23120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1990, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6101, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:09:54.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:09:54.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6253", 'reported_seq': 6315, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:52.132154+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:09:52.132154+0000', 'last_peered': '2024-09-15T03:09:52.132154+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:09:52.132154+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6253, 'log_dups_size': 0, 'ondisk_log_size': 6253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47150706, 'num_objects': 5756, 'num_object_clones': 0, 'num_object_copies': 23024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2084, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6251, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3674}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:09:54.398 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:09:55.707 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:55.707 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30524/365624 objects degraded (8.348%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":318,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91406,"data_bytes":748755540,"bytes_used":2643226624,"bytes_avail":383903830016,"bytes_total":386547056640,"degraded_objects":30524,"degraded_total":365624,"degraded_ratio":0.083484672778592214,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":113231,"recovering_keys_per_sec":0,"num_objects_recovered":181,"num_bytes_recovered":1359872,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":238},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:09:55.776 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:09:55.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:09:57.151 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:09:57.155 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:09:57.188 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-15T03:09:56.761096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":748755540,"num_objects":91406,"num_object_clones":0,"num_object_copies":365624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99301,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2158,"num_bytes_recovered":17326080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99328,"ondisk_log_size":99328,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2581276,"kb_used_data":1357100,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":374906084,"statfs":{"total":386547056640,"available":383903830016,"internally_reserved":0,"allocated":1389670400,"data_stored":1389004806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,13,4,9,8,4,3,0,0,0,0,7],"upper_bound":8192},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20348928,"num_objects":-2484,"num_object_clones":0,"num_object_copies":-9936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2484,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004875"},"pg_stats":[{"pgid":"1.f","version":"33'6178","reported_seq":6222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.411766+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:52.411766+0000","last_peered":"2024-09-15T03:09:52.411766+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:52.411766+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6178,"log_dups_size":0,"ondisk_log_size":6178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46956544,"num_objects":5732,"num_object_clones":0,"num_object_copies":22928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6178,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5948","reported_seq":6016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:51.948459+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:51.948459+0000","last_peered":"2024-09-15T03:09:51.948459+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:51.948459+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44873330,"num_objects":5478,"num_object_clones":0,"num_object_copies":21912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3482}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6346","reported_seq":6430,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.794786+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:52.794786+0000","last_peered":"2024-09-15T03:09:52.794786+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:52.794786+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48118609,"num_objects":5875,"num_object_clones":0,"num_object_copies":23500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6345,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3773}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6054","reported_seq":6378,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.126577+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:09:52.126577+0000","last_peered":"2024-09-15T03:09:52.126577+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:52.126577+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45659762,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1814,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6305","reported_seq":6363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.452922+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:52.452922+0000","last_peered":"2024-09-15T03:09:52.452922+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:52.452922+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6305,"log_dups_size":0,"ondisk_log_size":6305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47314546,"num_objects":5776,"num_object_clones":0,"num_object_copies":23104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6254","reported_seq":6300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.438858+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:52.438858+0000","last_peered":"2024-09-15T03:09:52.438858+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:52.438858+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47054848,"num_objects":5744,"num_object_clones":0,"num_object_copies":22976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6238","reported_seq":6311,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:51.776847+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:51.776847+0000","last_peered":"2024-09-15T03:09:51.776847+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:51.776847+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6238,"log_dups_size":0,"ondisk_log_size":6238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855794,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6236,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3671}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6280","reported_seq":6351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.038869+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:52.038869+0000","last_peered":"2024-09-15T03:09:52.038869+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:52.038869+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6280,"log_dups_size":0,"ondisk_log_size":6280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47552114,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6278,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3727}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6231","reported_seq":6290,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.266444+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:52.266444+0000","last_peered":"2024-09-15T03:09:52.266444+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:52.266444+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6231,"log_dups_size":0,"ondisk_log_size":6231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47008996,"num_objects":5739,"num_object_clones":0,"num_object_copies":22956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6248","reported_seq":6333,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.742240+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:52.742240+0000","last_peered":"2024-09-15T03:09:52.742240+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:52.742240+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6248,"log_dups_size":0,"ondisk_log_size":6248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46855794,"num_objects":5720,"num_object_clones":0,"num_object_copies":22880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6246,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3662}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6199","reported_seq":6261,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.259897+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:52.259897+0000","last_peered":"2024-09-15T03:09:52.259897+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:52.259897+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46559659,"num_objects":5684,"num_object_clones":0,"num_object_copies":22736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3607}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6197","reported_seq":10235,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:09:52.967646+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:09:52.967646+0000","last_peered":"2024-09-15T03:09:52.967646+0000","last_clean":"2024-09-15T03:09:52.967646+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:09:52.967646+0000","last_undegraded":"2024-09-15T03:09:52.967646+0000","last_fullsized":"2024-09-15T03:09:52.967646+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46709561,"num_objects":5702,"num_object_clones":0,"num_object_copies":22808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6337","reported_seq":6416,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.642496+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:52.642496+0000","last_peered":"2024-09-15T03:09:52.642496+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:52.642496+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47479609,"num_objects":5796,"num_object_clones":0,"num_object_copies":23184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6336,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3757}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6080","reported_seq":6132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.322275+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:52.322275+0000","last_peered":"2024-09-15T03:09:52.322275+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:52.322275+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45889138,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6078,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3566}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6180","reported_seq":6263,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.681144+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:52.681144+0000","last_peered":"2024-09-15T03:09:52.681144+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:52.681144+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6180,"log_dups_size":0,"ondisk_log_size":6180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46716530,"num_objects":5703,"num_object_clones":0,"num_object_copies":22812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1990,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6178,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3715}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6253","reported_seq":6315,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:52.132154+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:52.132154+0000","last_peered":"2024-09-15T03:09:52.132154+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:52.132154+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6253,"log_dups_size":0,"ondisk_log_size":6253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150706,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3674}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":748755540,"num_objects":91406,"num_object_clones":0,"num_object_copies":365624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91406,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99301,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2158,"num_bytes_recovered":17326080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1387073536,"data_stored":1387073536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99328,"ondisk_log_size":99328,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680416,"kb_used_data":365904,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93691424,"statfs":{"total":96636764160,"available":95940018176,"internally_reserved":0,"allocated":374685696,"data_stored":374518511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,4,1,1,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.34699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":0.315},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.46700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667868,"kb_used_data":366348,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93703972,"statfs":{"total":96636764160,"available":95952867328,"internally_reserved":0,"allocated":375140352,"data_stored":374976313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.20300000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.44500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.371},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.39600000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564428,"kb_used_data":258244,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93807412,"statfs":{"total":96636764160,"available":96058789888,"internally_reserved":0,"allocated":264441856,"data_stored":264274671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.80400000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.54800000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.58699999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668564,"kb_used_data":366604,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93703276,"statfs":{"total":96636764160,"available":95952154624,"internally_reserved":0,"allocated":375402496,"data_stored":375235311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,2,2,1,3,2,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.222},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.29299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.375},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":374644736,"data_stored":374644736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264142848,"data_stored":264142848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374001664,"data_stored":374001664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374284288,"data_stored":374284288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:00.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:01.701 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:01.737 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:01.760 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-15T03:10:00.764479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":739613268,"num_objects":90290,"num_object_clones":0,"num_object_copies":361160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90290,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100417,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":17907712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100444,"ondisk_log_size":100444,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2571820,"kb_used_data":1343500,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":374915540,"statfs":{"total":386547056640,"available":383913512960,"internally_reserved":0,"allocated":1375744000,"data_stored":1375078406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,7,5,12,5,1,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19226624,"num_objects":-2347,"num_object_clones":0,"num_object_copies":-9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007317"},"pg_stats":[{"pgid":"1.f","version":"33'6242","reported_seq":6286,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.542522+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:09:57.542522+0000","last_peered":"2024-09-15T03:09:57.542522+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:09:57.542522+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46432256,"num_objects":5668,"num_object_clones":0,"num_object_copies":22672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6020","reported_seq":6088,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.118943+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:09:57.118943+0000","last_peered":"2024-09-15T03:09:57.118943+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:09:57.118943+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44283506,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6018,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6424","reported_seq":6514,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.830591+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:57.830591+0000","last_peered":"2024-09-15T03:09:57.830591+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:57.830591+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47479633,"num_objects":5797,"num_object_clones":0,"num_object_copies":23188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6423,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3698}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6127","reported_seq":6593,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.141240+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:09:57.141240+0000","last_peered":"2024-09-15T03:09:57.141240+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:09:57.141240+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6127,"log_dups_size":0,"ondisk_log_size":6127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45061746,"num_objects":5501,"num_object_clones":0,"num_object_copies":22004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6125,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":1589248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3758}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6391","reported_seq":6453,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.383824+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:09:57.383824+0000","last_peered":"2024-09-15T03:09:57.383824+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:09:57.383824+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6391,"log_dups_size":0,"ondisk_log_size":6391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46610034,"num_objects":5690,"num_object_clones":0,"num_object_copies":22760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6389,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3585}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6330","reported_seq":6380,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.362519+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:09:57.362519+0000","last_peered":"2024-09-15T03:09:57.362519+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:09:57.362519+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6330,"log_dups_size":0,"ondisk_log_size":6330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46432256,"num_objects":5668,"num_object_clones":0,"num_object_copies":22672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6330,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6305","reported_seq":6380,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.045968+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:09:57.045968+0000","last_peered":"2024-09-15T03:09:57.045968+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:09:57.045968+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6305,"log_dups_size":0,"ondisk_log_size":6305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46306930,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3605}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6355","reported_seq":6425,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.005884+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:09:57.005884+0000","last_peered":"2024-09-15T03:09:57.005884+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:09:57.005884+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6355,"log_dups_size":0,"ondisk_log_size":6355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46937714,"num_objects":5730,"num_object_clones":0,"num_object_copies":22920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6353,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6298","reported_seq":6359,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.509196+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:09:57.509196+0000","last_peered":"2024-09-15T03:09:57.509196+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:09:57.509196+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6298,"log_dups_size":0,"ondisk_log_size":6298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46460132,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3631}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6315","reported_seq":6400,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.832072+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:57.832072+0000","last_peered":"2024-09-15T03:09:57.832072+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:57.832072+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46306930,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6271","reported_seq":6335,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.479849+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:09:57.479849+0000","last_peered":"2024-09-15T03:09:57.479849+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:09:57.479849+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6271,"log_dups_size":0,"ondisk_log_size":6271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45969835,"num_objects":5612,"num_object_clones":0,"num_object_copies":22448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6268,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6250","reported_seq":10289,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:09:57.858421+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:09:57.858421+0000","last_peered":"2024-09-15T03:09:57.858421+0000","last_clean":"2024-09-15T03:09:57.858421+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:09:57.858421+0000","last_undegraded":"2024-09-15T03:09:57.858421+0000","last_fullsized":"2024-09-15T03:09:57.858421+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6250,"log_dups_size":0,"ondisk_log_size":6250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46275385,"num_objects":5649,"num_object_clones":0,"num_object_copies":22596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6392","reported_seq":6471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.728806+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:57.728806+0000","last_peered":"2024-09-15T03:09:57.728806+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:57.728806+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47029049,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6143","reported_seq":6197,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.473838+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:09:57.473838+0000","last_peered":"2024-09-15T03:09:57.473838+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:09:57.473838+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6143,"log_dups_size":0,"ondisk_log_size":6143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45373042,"num_objects":5539,"num_object_clones":0,"num_object_copies":22156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3504}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6247","reported_seq":6332,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.842378+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:57.842378+0000","last_peered":"2024-09-15T03:09:57.842378+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:57.842378+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46167666,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6334","reported_seq":6402,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.067900+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:09:57.067900+0000","last_peered":"2024-09-15T03:09:57.067900+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:09:57.067900+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6334,"log_dups_size":0,"ondisk_log_size":6334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46487154,"num_objects":5675,"num_object_clones":0,"num_object_copies":22700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6332,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":739613268,"num_objects":90290,"num_object_clones":0,"num_object_copies":361160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90290,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100417,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2243,"num_bytes_recovered":17907712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1373118464,"data_stored":1373118464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100444,"ondisk_log_size":100444,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663408,"kb_used_data":361384,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93708432,"statfs":{"total":96636764160,"available":95957434368,"internally_reserved":0,"allocated":370057216,"data_stored":369890031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,2,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.189},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":4.0830000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":5.6390000000000002},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":0.25700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":5.7329999999999997},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.22600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671660,"kb_used_data":361828,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93700180,"statfs":{"total":96636764160,"available":95948984320,"internally_reserved":0,"allocated":370511872,"data_stored":370347833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.35599999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.379},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.40400000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.39100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568336,"kb_used_data":257992,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93803504,"statfs":{"total":96636764160,"available":96054788096,"internally_reserved":0,"allocated":264183808,"data_stored":264016623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.434},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.26900000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":12.385999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":2.3300000000000001},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":1.742}]}]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668416,"kb_used_data":362296,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93703424,"statfs":{"total":96636764160,"available":95952306176,"internally_reserved":0,"allocated":370991104,"data_stored":370823919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,1,0,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.188},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":8.4589999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":11.872999999999999},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.93000000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":370139136,"data_stored":370139136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263876608,"data_stored":263876608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369385472,"data_stored":369385472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":369717248,"data_stored":369717248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:01.762 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:03.143 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:03.178 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:03.207 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-15T03:10:02.765060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":732641876,"num_objects":89439,"num_object_clones":0,"num_object_copies":357756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101268,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2307,"num_bytes_recovered":18391040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101295,"ondisk_log_size":101295,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2575444,"kb_used_data":1334644,"kb_used_omap":14,"kb_used_meta":1240753,"kb_avail":374911916,"statfs":{"total":386547056640,"available":383909801984,"internally_reserved":0,"allocated":1366675456,"data_stored":1366009862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1270531099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,10,7,11,4,1,0,0,0,0,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26198016,"num_objects":-3198,"num_object_clones":0,"num_object_copies":-12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3198,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007397"},"pg_stats":[{"pgid":"1.f","version":"33'6325","reported_seq":6369,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.480000+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:02.480000+0000","last_peered":"2024-09-15T03:10:02.480000+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:02.480000+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45752320,"num_objects":5585,"num_object_clones":0,"num_object_copies":22340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6325,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6105","reported_seq":6173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.069658+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:02.069658+0000","last_peered":"2024-09-15T03:10:02.069658+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:02.069658+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43587186,"num_objects":5321,"num_object_clones":0,"num_object_copies":21284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6103,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6424","reported_seq":6514,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.830591+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:09:57.830591+0000","last_peered":"2024-09-15T03:09:57.830591+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:09:57.830591+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47479633,"num_objects":5797,"num_object_clones":0,"num_object_copies":23188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6423,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3698}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6197","reported_seq":6781,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.169628+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:02.169628+0000","last_peered":"2024-09-15T03:10:02.169628+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:02.169628+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44488306,"num_objects":5431,"num_object_clones":0,"num_object_copies":21724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6195,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":2072576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6468","reported_seq":6530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.508974+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:02.508974+0000","last_peered":"2024-09-15T03:10:02.508974+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:02.508974+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6468,"log_dups_size":0,"ondisk_log_size":6468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45979250,"num_objects":5613,"num_object_clones":0,"num_object_copies":22452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6466,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6396","reported_seq":6446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.306507+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:02.306507+0000","last_peered":"2024-09-15T03:10:02.306507+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:02.306507+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45891584,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6396,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6384","reported_seq":6459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.128633+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:02.128633+0000","last_peered":"2024-09-15T03:10:02.128633+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:02.128633+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6384,"log_dups_size":0,"ondisk_log_size":6384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45659762,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6382,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6438","reported_seq":6513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.150552+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:02.150552+0000","last_peered":"2024-09-15T03:10:02.150552+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:02.150552+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6438,"log_dups_size":0,"ondisk_log_size":6438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46257778,"num_objects":5647,"num_object_clones":0,"num_object_copies":22588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3571}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6387","reported_seq":6448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.483463+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:02.483463+0000","last_peered":"2024-09-15T03:10:02.483463+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:02.483463+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45731044,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6384,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6315","reported_seq":6400,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.832072+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:09:57.832072+0000","last_peered":"2024-09-15T03:09:57.832072+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:09:57.832072+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46306930,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3595}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6341","reported_seq":6405,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.354667+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:02.354667+0000","last_peered":"2024-09-15T03:10:02.354667+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:02.354667+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45396395,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6250","reported_seq":10289,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:09:57.858421+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:09:57.858421+0000","last_peered":"2024-09-15T03:09:57.858421+0000","last_clean":"2024-09-15T03:09:57.858421+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:09:57.858421+0000","last_undegraded":"2024-09-15T03:09:57.858421+0000","last_fullsized":"2024-09-15T03:09:57.858421+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6250,"log_dups_size":0,"ondisk_log_size":6250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46275385,"num_objects":5649,"num_object_clones":0,"num_object_copies":22596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6249,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6392","reported_seq":6471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.728806+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:09:57.728806+0000","last_peered":"2024-09-15T03:09:57.728806+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:09:57.728806+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47029049,"num_objects":5741,"num_object_clones":0,"num_object_copies":22964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6212","reported_seq":6266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.581460+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:02.581460+0000","last_peered":"2024-09-15T03:10:02.581460+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:02.581460+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6212,"log_dups_size":0,"ondisk_log_size":6212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44807794,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6210,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3435}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6247","reported_seq":6332,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:09:57.842378+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:09:57.842378+0000","last_peered":"2024-09-15T03:09:57.842378+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:09:57.842378+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6247,"log_dups_size":0,"ondisk_log_size":6247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46167666,"num_objects":5636,"num_object_clones":0,"num_object_copies":22544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6245,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6414","reported_seq":6489,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.086408+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:02.086408+0000","last_peered":"2024-09-15T03:10:02.086408+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:02.086408+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6414,"log_dups_size":0,"ondisk_log_size":6414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45831794,"num_objects":5595,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6412,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":732641876,"num_objects":89439,"num_object_clones":0,"num_object_copies":357756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101268,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2307,"num_bytes_recovered":18391040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1363214336,"data_stored":1363214336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101295,"ondisk_log_size":101295,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663604,"kb_used_data":357420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93708236,"statfs":{"total":96636764160,"available":95957233664,"internally_reserved":0,"allocated":365998080,"data_stored":365830895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,4,3,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.002}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":1.464},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":5.7149999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":1.427},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":1.1040000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671660,"kb_used_data":361828,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93700180,"statfs":{"total":96636764160,"available":95948984320,"internally_reserved":0,"allocated":370511872,"data_stored":370347833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.35599999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.379},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.40400000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.39100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568336,"kb_used_data":257992,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93803504,"statfs":{"total":96636764160,"available":96054788096,"internally_reserved":0,"allocated":264183808,"data_stored":264016623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.434},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.26900000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":12.385999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":2.3300000000000001},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":1.742}]}]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671844,"kb_used_data":357404,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93699996,"statfs":{"total":96636764160,"available":95948795904,"internally_reserved":0,"allocated":365981696,"data_stored":365814511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1,4,1,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.25700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":7.0460000000000003},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":6.7039999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.376}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":365199360,"data_stored":365199360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263876608,"data_stored":263876608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369385472,"data_stored":369385472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364752896,"data_stored":364752896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:03.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:10:03.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6325", 'reported_seq': 6369, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.480000+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:10:02.480000+0000', 'last_peered': '2024-09-15T03:10:02.480000+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:10:02.480000+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6325, 'log_dups_size': 0, 'ondisk_log_size': 6325, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45752320, 'num_objects': 5585, 'num_object_clones': 0, 'num_object_copies': 22340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6325, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3609}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:03.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:10:03.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6105", 'reported_seq': 6173, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.069658+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:10:02.069658+0000', 'last_peered': '2024-09-15T03:10:02.069658+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:10:02.069658+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6105, 'log_dups_size': 0, 'ondisk_log_size': 6105, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43587186, 'num_objects': 5321, 'num_object_clones': 0, 'num_object_copies': 21284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1998, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6103, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3325}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:03.208 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:10:03.208 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6424", 'reported_seq': 6514, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:57.830591+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:09:57.830591+0000', 'last_peered': '2024-09-15T03:09:57.830591+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:09:57.830591+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6424, 'log_dups_size': 0, 'ondisk_log_size': 6424, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47479633, 'num_objects': 5797, 'num_object_clones': 0, 'num_object_copies': 23188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2100, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5797, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6423, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3698}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6197", 'reported_seq': 6781, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.169628+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:10:02.169628+0000', 'last_peered': '2024-09-15T03:10:02.169628+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:10:02.169628+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6197, 'log_dups_size': 0, 'ondisk_log_size': 6197, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44488306, 'num_objects': 5431, 'num_object_clones': 0, 'num_object_copies': 21724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1684, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6195, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 261, 'num_bytes_recovered': 2072576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6468", 'reported_seq': 6530, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.508974+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:10:02.508974+0000', 'last_peered': '2024-09-15T03:10:02.508974+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:10:02.508974+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6468, 'log_dups_size': 0, 'ondisk_log_size': 6468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45979250, 'num_objects': 5613, 'num_object_clones': 0, 'num_object_copies': 22452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2107, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6466, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3508}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6396", 'reported_seq': 6446, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.306507+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:10:02.306507+0000', 'last_peered': '2024-09-15T03:10:02.306507+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:10:02.306507+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6396, 'log_dups_size': 0, 'ondisk_log_size': 6396, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45891584, 'num_objects': 5602, 'num_object_clones': 0, 'num_object_copies': 22408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6396, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3548}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6384", 'reported_seq': 6459, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.128633+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:10:02.128633+0000', 'last_peered': '2024-09-15T03:10:02.128633+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:10:02.128633+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6384, 'log_dups_size': 0, 'ondisk_log_size': 6384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45659762, 'num_objects': 5574, 'num_object_clones': 0, 'num_object_copies': 22296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6382, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3526}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:10:03.209 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6438", 'reported_seq': 6513, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.150552+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:10:02.150552+0000', 'last_peered': '2024-09-15T03:10:02.150552+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:10:02.150552+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6438, 'log_dups_size': 0, 'ondisk_log_size': 6438, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46257778, 'num_objects': 5647, 'num_object_clones': 0, 'num_object_copies': 22588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6436, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3571}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6387", 'reported_seq': 6448, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.483463+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:10:02.483463+0000', 'last_peered': '2024-09-15T03:10:02.483463+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:10:02.483463+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6387, 'log_dups_size': 0, 'ondisk_log_size': 6387, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45731044, 'num_objects': 5583, 'num_object_clones': 0, 'num_object_copies': 22332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6384, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3542}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6315", 'reported_seq': 6400, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:57.832072+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:09:57.832072+0000', 'last_peered': '2024-09-15T03:09:57.832072+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:09:57.832072+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6315, 'log_dups_size': 0, 'ondisk_log_size': 6315, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46306930, 'num_objects': 5653, 'num_object_clones': 0, 'num_object_copies': 22612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6313, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3595}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6341", 'reported_seq': 6405, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.354667+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:10:02.354667+0000', 'last_peered': '2024-09-15T03:10:02.354667+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:10:02.354667+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6341, 'log_dups_size': 0, 'ondisk_log_size': 6341, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45396395, 'num_objects': 5542, 'num_object_clones': 0, 'num_object_copies': 22168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6338, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3466}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6392", 'reported_seq': 6471, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:57.728806+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:09:57.728806+0000', 'last_peered': '2024-09-15T03:09:57.728806+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:09:57.728806+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6392, 'log_dups_size': 0, 'ondisk_log_size': 6392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47029049, 'num_objects': 5741, 'num_object_clones': 0, 'num_object_copies': 22964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6391, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6212", 'reported_seq': 6266, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.581460+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:10:02.581460+0000', 'last_peered': '2024-09-15T03:10:02.581460+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:10:02.581460+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6212, 'log_dups_size': 0, 'ondisk_log_size': 6212, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44807794, 'num_objects': 5470, 'num_object_clones': 0, 'num_object_copies': 21880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6210, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3435}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:03.210 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:10:03.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6247", 'reported_seq': 6332, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:09:57.842378+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:09:57.842378+0000', 'last_peered': '2024-09-15T03:09:57.842378+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:09:57.842378+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6247, 'log_dups_size': 0, 'ondisk_log_size': 6247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46167666, 'num_objects': 5636, 'num_object_clones': 0, 'num_object_copies': 22544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6245, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:03.211 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:10:03.211 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6414", 'reported_seq': 6489, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:02.086408+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:10:02.086408+0000', 'last_peered': '2024-09-15T03:10:02.086408+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:10:02.086408+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6414, 'log_dups_size': 0, 'ondisk_log_size': 6414, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45831794, 'num_objects': 5595, 'num_object_clones': 0, 'num_object_copies': 22380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6412, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:03.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:10:04.988 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:04.989 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30375/357756 objects degraded (8.490%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":328,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89439,"data_bytes":732641876,"bytes_used":2637254656,"bytes_avail":383909801984,"bytes_total":386547056640,"degraded_objects":30375,"degraded_total":357756,"degraded_ratio":0.084904236406936576,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":125533,"recovering_keys_per_sec":0,"num_objects_recovered":213,"num_bytes_recovered":1507328,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":266},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:10:05.032 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:10:05.033 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:06.349 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:06.374 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-15T03:10:04.765569+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":729651796,"num_objects":89074,"num_object_clones":0,"num_object_copies":356296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89074,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101633,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2307,"num_bytes_recovered":18391040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101660,"ondisk_log_size":101660,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2545452,"kb_used_data":1329576,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":374941908,"statfs":{"total":386547056640,"available":383940513792,"internally_reserved":0,"allocated":1361485824,"data_stored":1360820230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,6,9,6,9,4,1,0,0,0,0,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22249472,"num_objects":-2716,"num_object_clones":0,"num_object_copies":-10864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2716,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005480"},"pg_stats":[{"pgid":"1.f","version":"33'6325","reported_seq":6369,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.480000+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:02.480000+0000","last_peered":"2024-09-15T03:10:02.480000+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:02.480000+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45752320,"num_objects":5585,"num_object_clones":0,"num_object_copies":22340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6325,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6105","reported_seq":6173,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.069658+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:02.069658+0000","last_peered":"2024-09-15T03:10:02.069658+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:02.069658+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43587186,"num_objects":5321,"num_object_clones":0,"num_object_copies":21284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6103,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6503","reported_seq":6593,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.771935+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:02.771935+0000","last_peered":"2024-09-15T03:10:02.771935+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:02.771935+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6503,"log_dups_size":0,"ondisk_log_size":6503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46832465,"num_objects":5718,"num_object_clones":0,"num_object_copies":22872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6502,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3619}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6197","reported_seq":6781,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.169628+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:02.169628+0000","last_peered":"2024-09-15T03:10:02.169628+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:02.169628+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44488306,"num_objects":5431,"num_object_clones":0,"num_object_copies":21724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6195,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":261,"num_bytes_recovered":2072576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6468","reported_seq":6530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.508974+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:02.508974+0000","last_peered":"2024-09-15T03:10:02.508974+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:02.508974+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6468,"log_dups_size":0,"ondisk_log_size":6468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45979250,"num_objects":5613,"num_object_clones":0,"num_object_copies":22452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6466,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6396","reported_seq":6446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.306507+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:02.306507+0000","last_peered":"2024-09-15T03:10:02.306507+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:02.306507+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45891584,"num_objects":5602,"num_object_clones":0,"num_object_copies":22408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6396,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3548}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6384","reported_seq":6459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.128633+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:02.128633+0000","last_peered":"2024-09-15T03:10:02.128633+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:02.128633+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6384,"log_dups_size":0,"ondisk_log_size":6384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45659762,"num_objects":5574,"num_object_clones":0,"num_object_copies":22296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6382,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3526}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6438","reported_seq":6513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.150552+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:02.150552+0000","last_peered":"2024-09-15T03:10:02.150552+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:02.150552+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6438,"log_dups_size":0,"ondisk_log_size":6438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46257778,"num_objects":5647,"num_object_clones":0,"num_object_copies":22588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3571}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6387","reported_seq":6448,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.483463+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:02.483463+0000","last_peered":"2024-09-15T03:10:02.483463+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:02.483463+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45731044,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6384,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6388","reported_seq":6472,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.810957+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:02.810957+0000","last_peered":"2024-09-15T03:10:02.810957+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:02.810957+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6388,"log_dups_size":0,"ondisk_log_size":6388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708914,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6386,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6341","reported_seq":6405,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.354667+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:02.354667+0000","last_peered":"2024-09-15T03:10:02.354667+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:02.354667+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45396395,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6338,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6321","reported_seq":10360,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:03.110283+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:03.110283+0000","last_peered":"2024-09-15T03:10:03.110283+0000","last_clean":"2024-09-15T03:10:03.110283+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:03.110283+0000","last_undegraded":"2024-09-15T03:10:03.110283+0000","last_fullsized":"2024-09-15T03:10:03.110283+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45693753,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6465","reported_seq":6544,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.777006+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:02.777006+0000","last_peered":"2024-09-15T03:10:02.777006+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:02.777006+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46431033,"num_objects":5668,"num_object_clones":0,"num_object_copies":22672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6464,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6212","reported_seq":6266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.581460+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:02.581460+0000","last_peered":"2024-09-15T03:10:02.581460+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:02.581460+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6212,"log_dups_size":0,"ondisk_log_size":6212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44807794,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6210,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3435}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6316","reported_seq":6401,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.795352+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:02.795352+0000","last_peered":"2024-09-15T03:10:02.795352+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:02.795352+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45602418,"num_objects":5567,"num_object_clones":0,"num_object_copies":22268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3580}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6414","reported_seq":6489,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:02.086408+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:02.086408+0000","last_peered":"2024-09-15T03:10:02.086408+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:02.086408+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6414,"log_dups_size":0,"ondisk_log_size":6414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45831794,"num_objects":5595,"num_object_clones":0,"num_object_copies":22380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6412,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":729651796,"num_objects":89074,"num_object_clones":0,"num_object_copies":356296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89074,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101633,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2307,"num_bytes_recovered":18391040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1358020608,"data_stored":1358020608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101660,"ondisk_log_size":101660,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663604,"kb_used_data":357420,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93708236,"statfs":{"total":96636764160,"available":95957233664,"internally_reserved":0,"allocated":365998080,"data_stored":365830895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,4,3,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.002}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":1.464},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":5.7149999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":1.427},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":1.1040000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654432,"kb_used_data":357044,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93717408,"statfs":{"total":96636764160,"available":95966625792,"internally_reserved":0,"allocated":365613056,"data_stored":365449017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,4,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.254},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.20300000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":2.1269999999999998},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":2.149}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":2.0859999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.35899999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555572,"kb_used_data":257708,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93816268,"statfs":{"total":96636764160,"available":96067858432,"internally_reserved":0,"allocated":263892992,"data_stored":263725807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.57399999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":0.378},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":0.24299999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":10.680999999999999},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.59599999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671844,"kb_used_data":357404,"kb_used_omap":4,"kb_used_meta":314427,"kb_avail":93699996,"statfs":{"total":96636764160,"available":95948795904,"internally_reserved":0,"allocated":365981696,"data_stored":365814511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":321973825},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1,4,1,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.25700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":7.0460000000000003},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":6.7039999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.376}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":365199360,"data_stored":365199360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263585792,"data_stored":263585792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364482560,"data_stored":364482560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364752896,"data_stored":364752896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:09.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:10.707 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:10.738 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:10.826 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-15T03:10:08.766548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":719829588,"num_objects":87875,"num_object_clones":0,"num_object_copies":351500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102832,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2369,"num_bytes_recovered":18817024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102859,"ondisk_log_size":102859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2539312,"kb_used_data":1315152,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":374948048,"statfs":{"total":386547056640,"available":383946801152,"internally_reserved":0,"allocated":1346715648,"data_stored":1346050054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,11,8,13,5,1,0,0,1,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28925952,"num_objects":-3531,"num_object_clones":0,"num_object_copies":-14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3531,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":1490944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005451"},"pg_stats":[{"pgid":"1.f","version":"33'6402","reported_seq":6450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.427621+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:07.427621+0000","last_peered":"2024-09-15T03:10:07.427621+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:07.427621+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45121536,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3534}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6177","reported_seq":6247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.108065+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:07.108065+0000","last_peered":"2024-09-15T03:10:07.108065+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:07.108065+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42997362,"num_objects":5249,"num_object_clones":0,"num_object_copies":20996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6175,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6588","reported_seq":6682,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.754788+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:07.754788+0000","last_peered":"2024-09-15T03:10:07.754788+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:07.754788+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46136145,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6587,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6265","reported_seq":6953,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.178191+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:07.178191+0000","last_peered":"2024-09-15T03:10:07.178191+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:07.178191+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6265,"log_dups_size":0,"ondisk_log_size":6265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43931250,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6263,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":2498560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6543","reported_seq":6609,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.613494+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:07.613494+0000","last_peered":"2024-09-15T03:10:07.613494+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:07.613494+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45364850,"num_objects":5538,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6541,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6465","reported_seq":6517,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.476985+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:07.476985+0000","last_peered":"2024-09-15T03:10:07.476985+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:07.476985+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45326336,"num_objects":5533,"num_object_clones":0,"num_object_copies":22132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6465,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6445","reported_seq":6520,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:06.986418+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:06.986418+0000","last_peered":"2024-09-15T03:10:06.986418+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:06.986418+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45160050,"num_objects":5513,"num_object_clones":0,"num_object_copies":22052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3465}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6507","reported_seq":6582,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.188410+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:07.188410+0000","last_peered":"2024-09-15T03:10:07.188410+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:07.188410+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6507,"log_dups_size":0,"ondisk_log_size":6507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45692530,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6505,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6476","reported_seq":6539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.575994+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:07.575994+0000","last_peered":"2024-09-15T03:10:07.575994+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:07.575994+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6476,"log_dups_size":0,"ondisk_log_size":6476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45001956,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6465","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.854683+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:07.854683+0000","last_peered":"2024-09-15T03:10:07.854683+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:07.854683+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45078130,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6463,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3445}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6413","reported_seq":6477,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.524629+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:07.524629+0000","last_peered":"2024-09-15T03:10:07.524629+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:07.524629+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6413,"log_dups_size":0,"ondisk_log_size":6413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44806571,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6410,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6396","reported_seq":10434,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:08.094344+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:08.094344+0000","last_peered":"2024-09-15T03:10:08.094344+0000","last_clean":"2024-09-15T03:10:08.094344+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:08.094344+0000","last_undegraded":"2024-09-15T03:10:08.094344+0000","last_fullsized":"2024-09-15T03:10:08.094344+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45079353,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6395,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6543","reported_seq":6624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.903454+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:07.903454+0000","last_peered":"2024-09-15T03:10:07.903454+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:07.903454+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45792057,"num_objects":5590,"num_object_clones":0,"num_object_copies":22360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6289","reported_seq":6343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.337050+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:07.337050+0000","last_peered":"2024-09-15T03:10:07.337050+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:07.337050+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44177010,"num_objects":5393,"num_object_clones":0,"num_object_copies":21572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6393","reported_seq":6478,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.778204+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:07.778204+0000","last_peered":"2024-09-15T03:10:07.778204+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:07.778204+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6393,"log_dups_size":0,"ondisk_log_size":6393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44971634,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6492","reported_seq":6567,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.042810+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:07.042810+0000","last_peered":"2024-09-15T03:10:07.042810+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:07.042810+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6492,"log_dups_size":0,"ondisk_log_size":6492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45192818,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6490,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3440}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":719829588,"num_objects":87875,"num_object_clones":0,"num_object_copies":351500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102832,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2369,"num_bytes_recovered":18817024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1343021056,"data_stored":1343021056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102859,"ondisk_log_size":102859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667136,"kb_used_data":352632,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93704704,"statfs":{"total":96636764160,"available":95953616896,"internally_reserved":0,"allocated":361095168,"data_stored":360927983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,4,3,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.002}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":1.464},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":5.7149999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":1.427},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":1.1040000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653832,"kb_used_data":352320,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93718008,"statfs":{"total":96636764160,"available":95967240192,"internally_reserved":0,"allocated":360775680,"data_stored":360611641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,2,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.251}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.45100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.435}]}]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563640,"kb_used_data":257456,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93808200,"statfs":{"total":96636764160,"available":96059596800,"internally_reserved":0,"allocated":263634944,"data_stored":263467759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.23400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":1.381},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":4.7290000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":1.103},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":1.349}]}]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654704,"kb_used_data":352744,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93717136,"statfs":{"total":96636764160,"available":95966347264,"internally_reserved":0,"allocated":361209856,"data_stored":361042671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.193},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.40300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":2.0960000000000001},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.50900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.34899999999999998},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.375}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":360296448,"data_stored":360296448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263311360,"data_stored":263311360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359534592,"data_stored":359534592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359878656,"data_stored":359878656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:10.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:11.868 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:11.890 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:11.942 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-15T03:10:10.767101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":719829588,"num_objects":87875,"num_object_clones":0,"num_object_copies":351500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102832,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2369,"num_bytes_recovered":18817024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102859,"ondisk_log_size":102859,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2539312,"kb_used_data":1315152,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":374948048,"statfs":{"total":386547056640,"available":383946801152,"internally_reserved":0,"allocated":1346715648,"data_stored":1346050054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,11,8,13,5,1,0,0,1,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19783680,"num_objects":-2415,"num_object_clones":0,"num_object_copies":-9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2415,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":909312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005533"},"pg_stats":[{"pgid":"1.f","version":"33'6402","reported_seq":6450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.427621+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:07.427621+0000","last_peered":"2024-09-15T03:10:07.427621+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:07.427621+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45121536,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3534}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6177","reported_seq":6247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.108065+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:07.108065+0000","last_peered":"2024-09-15T03:10:07.108065+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:07.108065+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42997362,"num_objects":5249,"num_object_clones":0,"num_object_copies":20996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6175,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6588","reported_seq":6682,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.754788+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:07.754788+0000","last_peered":"2024-09-15T03:10:07.754788+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:07.754788+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6588,"log_dups_size":0,"ondisk_log_size":6588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46136145,"num_objects":5633,"num_object_clones":0,"num_object_copies":22532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6587,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6265","reported_seq":6953,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.178191+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:07.178191+0000","last_peered":"2024-09-15T03:10:07.178191+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:07.178191+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6265,"log_dups_size":0,"ondisk_log_size":6265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43931250,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6263,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":2498560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6543","reported_seq":6609,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.613494+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:07.613494+0000","last_peered":"2024-09-15T03:10:07.613494+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:07.613494+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45364850,"num_objects":5538,"num_object_clones":0,"num_object_copies":22152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6541,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6465","reported_seq":6517,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.476985+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:07.476985+0000","last_peered":"2024-09-15T03:10:07.476985+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:07.476985+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45326336,"num_objects":5533,"num_object_clones":0,"num_object_copies":22132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6465,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6445","reported_seq":6520,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:06.986418+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:06.986418+0000","last_peered":"2024-09-15T03:10:06.986418+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:06.986418+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45160050,"num_objects":5513,"num_object_clones":0,"num_object_copies":22052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3465}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6507","reported_seq":6582,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.188410+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:07.188410+0000","last_peered":"2024-09-15T03:10:07.188410+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:07.188410+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6507,"log_dups_size":0,"ondisk_log_size":6507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45692530,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6505,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3502}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6476","reported_seq":6539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.575994+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:07.575994+0000","last_peered":"2024-09-15T03:10:07.575994+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:07.575994+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6476,"log_dups_size":0,"ondisk_log_size":6476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45001956,"num_objects":5494,"num_object_clones":0,"num_object_copies":21976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6465","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.854683+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:07.854683+0000","last_peered":"2024-09-15T03:10:07.854683+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:07.854683+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6465,"log_dups_size":0,"ondisk_log_size":6465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45078130,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6463,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3445}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6413","reported_seq":6477,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.524629+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:07.524629+0000","last_peered":"2024-09-15T03:10:07.524629+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:07.524629+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6413,"log_dups_size":0,"ondisk_log_size":6413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44806571,"num_objects":5470,"num_object_clones":0,"num_object_copies":21880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6410,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6396","reported_seq":10434,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:08.094344+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:08.094344+0000","last_peered":"2024-09-15T03:10:08.094344+0000","last_clean":"2024-09-15T03:10:08.094344+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:08.094344+0000","last_undegraded":"2024-09-15T03:10:08.094344+0000","last_fullsized":"2024-09-15T03:10:08.094344+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45079353,"num_objects":5503,"num_object_clones":0,"num_object_copies":22012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6395,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6543","reported_seq":6624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.903454+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:07.903454+0000","last_peered":"2024-09-15T03:10:07.903454+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:07.903454+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45792057,"num_objects":5590,"num_object_clones":0,"num_object_copies":22360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6542,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3552}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6289","reported_seq":6343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.337050+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:07.337050+0000","last_peered":"2024-09-15T03:10:07.337050+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:07.337050+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44177010,"num_objects":5393,"num_object_clones":0,"num_object_copies":21572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3358}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6393","reported_seq":6478,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.778204+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:07.778204+0000","last_peered":"2024-09-15T03:10:07.778204+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:07.778204+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6393,"log_dups_size":0,"ondisk_log_size":6393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44971634,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6391,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6492","reported_seq":6567,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:07.042810+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:07.042810+0000","last_peered":"2024-09-15T03:10:07.042810+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:07.042810+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6492,"log_dups_size":0,"ondisk_log_size":6492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45192818,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6490,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3440}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":719829588,"num_objects":87875,"num_object_clones":0,"num_object_copies":351500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102832,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2369,"num_bytes_recovered":18817024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1343021056,"data_stored":1343021056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102859,"ondisk_log_size":102859,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667136,"kb_used_data":352632,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93704704,"statfs":{"total":96636764160,"available":95953616896,"internally_reserved":0,"allocated":361095168,"data_stored":360927983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,4,3,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.002}]},{"osd":1,"last update":"Sun Sep 15 03:09:11 2024","interfaces":[{"interface":"back","average":{"1min":2.3079999999999998,"5min":1.788,"15min":1.6839999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":13.31,"5min":13.31,"15min":13.31},"last":1.464},{"interface":"front","average":{"1min":2.7189999999999999,"5min":1.4319999999999999,"15min":1.2629999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.279999999999999,"5min":13.279999999999999,"15min":13.279999999999999},"last":5.7149999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":1.427},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":1.1040000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":653832,"kb_used_data":352320,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93718008,"statfs":{"total":96636764160,"available":95967240192,"internally_reserved":0,"allocated":360775680,"data_stored":360611641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,2,1,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.251}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.45100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.435}]}]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563640,"kb_used_data":257456,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93808200,"statfs":{"total":96636764160,"available":96059596800,"internally_reserved":0,"allocated":263634944,"data_stored":263467759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":1.6000000000000001,"15min":2.044},"min":{"1min":0.188,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.02,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.698,"5min":1.5760000000000001,"15min":1.49},"min":{"1min":0.19900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.23400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.3460000000000001,"15min":1.3859999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":10.433,"5min":10.433,"15min":10.433},"last":1.381},{"interface":"front","average":{"1min":1.167,"5min":1.417,"15min":1.633},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":6.0730000000000004,"5min":8.3859999999999992,"15min":8.3859999999999992},"last":4.7290000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.423,"15min":1.5800000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.8929999999999998,"5min":13.414999999999999,"15min":13.414999999999999},"last":1.103},{"interface":"front","average":{"1min":1.7909999999999999,"5min":1.371,"15min":1.272},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":13.973000000000001,"5min":13.973000000000001,"15min":13.973000000000001},"last":1.349}]}]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654704,"kb_used_data":352744,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93717136,"statfs":{"total":96636764160,"available":95966347264,"internally_reserved":0,"allocated":361209856,"data_stored":361042671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.193},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":0.40300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":2.0960000000000001},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.50900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.34899999999999998},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.375}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":360296448,"data_stored":360296448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263311360,"data_stored":263311360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359534592,"data_stored":359534592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359878656,"data_stored":359878656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:11.943 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:10:11.943 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6402", 'reported_seq': 6450, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.427621+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:10:07.427621+0000', 'last_peered': '2024-09-15T03:10:07.427621+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:10:07.427621+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6402, 'log_dups_size': 0, 'ondisk_log_size': 6402, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45121536, 'num_objects': 5508, 'num_object_clones': 0, 'num_object_copies': 22032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1974, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6402, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3534}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6177", 'reported_seq': 6247, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.108065+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:10:07.108065+0000', 'last_peered': '2024-09-15T03:10:07.108065+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:10:07.108065+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6177, 'log_dups_size': 0, 'ondisk_log_size': 6177, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42997362, 'num_objects': 5249, 'num_object_clones': 0, 'num_object_copies': 20996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6175, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6588", 'reported_seq': 6682, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.754788+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:10:07.754788+0000', 'last_peered': '2024-09-15T03:10:07.754788+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:10:07.754788+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6588, 'log_dups_size': 0, 'ondisk_log_size': 6588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46136145, 'num_objects': 5633, 'num_object_clones': 0, 'num_object_copies': 22532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5633, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6587, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3536}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6265", 'reported_seq': 6953, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.178191+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:10:07.178191+0000', 'last_peered': '2024-09-15T03:10:07.178191+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:10:07.178191+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6265, 'log_dups_size': 0, 'ondisk_log_size': 6265, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43931250, 'num_objects': 5363, 'num_object_clones': 0, 'num_object_copies': 21452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1632, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6263, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 313, 'num_bytes_recovered': 2498560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6543", 'reported_seq': 6609, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.613494+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:10:07.613494+0000', 'last_peered': '2024-09-15T03:10:07.613494+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:10:07.613494+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6543, 'log_dups_size': 0, 'ondisk_log_size': 6543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45364850, 'num_objects': 5538, 'num_object_clones': 0, 'num_object_copies': 22152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2105, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6541, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3435}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:11.944 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6465", 'reported_seq': 6517, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.476985+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:10:07.476985+0000', 'last_peered': '2024-09-15T03:10:07.476985+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:10:07.476985+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6465, 'log_dups_size': 0, 'ondisk_log_size': 6465, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45326336, 'num_objects': 5533, 'num_object_clones': 0, 'num_object_copies': 22132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2053, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6465, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3480}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6445", 'reported_seq': 6520, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:06.986418+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:10:06.986418+0000', 'last_peered': '2024-09-15T03:10:06.986418+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:10:06.986418+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6445, 'log_dups_size': 0, 'ondisk_log_size': 6445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45160050, 'num_objects': 5513, 'num_object_clones': 0, 'num_object_copies': 22052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6443, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3465}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6507", 'reported_seq': 6582, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.188410+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:10:07.188410+0000', 'last_peered': '2024-09-15T03:10:07.188410+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:10:07.188410+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6507, 'log_dups_size': 0, 'ondisk_log_size': 6507, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45692530, 'num_objects': 5578, 'num_object_clones': 0, 'num_object_copies': 22312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6505, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6476", 'reported_seq': 6539, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.575994+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:10:07.575994+0000', 'last_peered': '2024-09-15T03:10:07.575994+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:10:07.575994+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6476, 'log_dups_size': 0, 'ondisk_log_size': 6476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45001956, 'num_objects': 5494, 'num_object_clones': 0, 'num_object_copies': 21976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2043, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6473, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3454}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6465", 'reported_seq': 6550, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.854683+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:10:07.854683+0000', 'last_peered': '2024-09-15T03:10:07.854683+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:10:07.854683+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6465, 'log_dups_size': 0, 'ondisk_log_size': 6465, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45078130, 'num_objects': 5503, 'num_object_clones': 0, 'num_object_copies': 22012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6463, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3445}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:11.945 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:10:11.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6413", 'reported_seq': 6477, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.524629+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:10:07.524629+0000', 'last_peered': '2024-09-15T03:10:07.524629+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:10:07.524629+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6413, 'log_dups_size': 0, 'ondisk_log_size': 6413, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44806571, 'num_objects': 5470, 'num_object_clones': 0, 'num_object_copies': 21880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6410, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:11.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:10:11.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6543", 'reported_seq': 6624, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.903454+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:10:07.903454+0000', 'last_peered': '2024-09-15T03:10:07.903454+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:10:07.903454+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6543, 'log_dups_size': 0, 'ondisk_log_size': 6543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45792057, 'num_objects': 5590, 'num_object_clones': 0, 'num_object_copies': 22360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6542, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:11.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:10:11.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6289", 'reported_seq': 6343, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.337050+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:10:07.337050+0000', 'last_peered': '2024-09-15T03:10:07.337050+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:10:07.337050+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6289, 'log_dups_size': 0, 'ondisk_log_size': 6289, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44177010, 'num_objects': 5393, 'num_object_clones': 0, 'num_object_copies': 21572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5393, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6287, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:11.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:10:11.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6393", 'reported_seq': 6478, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.778204+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:10:07.778204+0000', 'last_peered': '2024-09-15T03:10:07.778204+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:10:07.778204+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6393, 'log_dups_size': 0, 'ondisk_log_size': 6393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44971634, 'num_objects': 5490, 'num_object_clones': 0, 'num_object_copies': 21960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6391, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:11.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:10:11.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6492", 'reported_seq': 6567, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:07.042810+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:10:07.042810+0000', 'last_peered': '2024-09-15T03:10:07.042810+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:10:07.042810+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6492, 'log_dups_size': 0, 'ondisk_log_size': 6492, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45192818, 'num_objects': 5517, 'num_object_clones': 0, 'num_object_copies': 22068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6490, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3440}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:11.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:10:13.563 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:13.563 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30240/348272 objects degraded (8.683%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":336,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":87068,"data_bytes":713218644,"bytes_used":2582011904,"bytes_avail":383965044736,"bytes_total":386547056640,"degraded_objects":30240,"degraded_total":348272,"degraded_ratio":0.086828685625028712,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":125578,"recovering_keys_per_sec":0,"num_objects_recovered":199,"num_bytes_recovered":1507328,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":268},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:10:13.610 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:10:13.611 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:14.962 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:15.014 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:15.048 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-15T03:10:14.768110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":710326868,"num_objects":86715,"num_object_clones":0,"num_object_copies":346860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86715,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103992,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2442,"num_bytes_recovered":19415040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104019,"ondisk_log_size":104019,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2528968,"kb_used_data":1300648,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":374958392,"statfs":{"total":386547056640,"available":383957393408,"internally_reserved":0,"allocated":1331863552,"data_stored":1331197958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,5,7,7,5,0,0,0,0,0,0,5],"upper_bound":16384},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22315008,"num_objects":-2724,"num_object_clones":0,"num_object_copies":-10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2724,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003049"},"pg_stats":[{"pgid":"1.f","version":"33'6466","reported_seq":6514,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.433584+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:12.433584+0000","last_peered":"2024-09-15T03:10:12.433584+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:12.433584+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6466,"log_dups_size":0,"ondisk_log_size":6466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44597248,"num_objects":5444,"num_object_clones":0,"num_object_copies":21776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6466,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3470}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6243","reported_seq":6313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:11.974748+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:11.974748+0000","last_peered":"2024-09-15T03:10:11.974748+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:11.974748+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6243,"log_dups_size":0,"ondisk_log_size":6243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42456690,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6241,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6668","reported_seq":6762,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.714869+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:12.714869+0000","last_peered":"2024-09-15T03:10:12.714869+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:12.714869+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6668,"log_dups_size":0,"ondisk_log_size":6668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45480785,"num_objects":5553,"num_object_clones":0,"num_object_copies":22212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6667,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3456}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6326","reported_seq":7160,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.245175+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:12.245175+0000","last_peered":"2024-09-15T03:10:12.245175+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:12.245175+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43431538,"num_objects":5302,"num_object_clones":0,"num_object_copies":21208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1559,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":3096576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6617","reported_seq":6683,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.495802+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:12.495802+0000","last_peered":"2024-09-15T03:10:12.495802+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:12.495802+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44758642,"num_objects":5464,"num_object_clones":0,"num_object_copies":21856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3361}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6553","reported_seq":6605,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.547362+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:12.547362+0000","last_peered":"2024-09-15T03:10:12.547362+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:12.547362+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6553,"log_dups_size":0,"ondisk_log_size":6553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44605440,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2053,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6553,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3392}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6537","reported_seq":6612,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.142315+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:12.142315+0000","last_peered":"2024-09-15T03:10:12.142315+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:12.142315+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6537,"log_dups_size":0,"ondisk_log_size":6537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44406386,"num_objects":5421,"num_object_clones":0,"num_object_copies":21684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6535,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3373}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6594","reported_seq":6669,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.208264+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:12.208264+0000","last_peered":"2024-09-15T03:10:12.208264+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:12.208264+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44979826,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6592,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6550","reported_seq":6613,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.514468+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:12.514468+0000","last_peered":"2024-09-15T03:10:12.514468+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:12.514468+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44395748,"num_objects":5420,"num_object_clones":0,"num_object_copies":21680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6547,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3380}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6526","reported_seq":6611,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.861465+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:12.861465+0000","last_peered":"2024-09-15T03:10:12.861465+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:12.861465+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6526,"log_dups_size":0,"ondisk_log_size":6526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44578418,"num_objects":5442,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6524,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3384}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6481","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.529749+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:12.529749+0000","last_peered":"2024-09-15T03:10:12.529749+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:12.529749+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6481,"log_dups_size":0,"ondisk_log_size":6481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44249515,"num_objects":5402,"num_object_clones":0,"num_object_copies":21608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6478,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6467","reported_seq":10506,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:12.924054+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:12.924054+0000","last_peered":"2024-09-15T03:10:12.924054+0000","last_clean":"2024-09-15T03:10:12.924054+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:12.924054+0000","last_undegraded":"2024-09-15T03:10:12.924054+0000","last_fullsized":"2024-09-15T03:10:12.924054+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6467,"log_dups_size":0,"ondisk_log_size":6467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44497721,"num_objects":5432,"num_object_clones":0,"num_object_copies":21728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6466,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6608","reported_seq":6689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.939970+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:12.939970+0000","last_peered":"2024-09-15T03:10:12.939970+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:12.939970+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6608,"log_dups_size":0,"ondisk_log_size":6608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45259577,"num_objects":5525,"num_object_clones":0,"num_object_copies":22100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6607,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6354","reported_seq":6408,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.402390+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:12.402390+0000","last_peered":"2024-09-15T03:10:12.402390+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:12.402390+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43644530,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3293}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6469","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.813786+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:12.813786+0000","last_peered":"2024-09-15T03:10:12.813786+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:12.813786+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44349042,"num_objects":5414,"num_object_clones":0,"num_object_copies":21656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3427}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6560","reported_seq":6635,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:12.237800+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:12.237800+0000","last_peered":"2024-09-15T03:10:12.237800+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:12.237800+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6560,"log_dups_size":0,"ondisk_log_size":6560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44635762,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6558,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3372}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":710326868,"num_objects":86715,"num_object_clones":0,"num_object_copies":346860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86715,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103992,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2442,"num_bytes_recovered":19415040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1328377856,"data_stored":1328377856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104019,"ondisk_log_size":104019,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649908,"kb_used_data":347884,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93721932,"statfs":{"total":96636764160,"available":95971258368,"internally_reserved":0,"allocated":356233216,"data_stored":356066031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,4,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.441}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.46300000000000002},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":15.648999999999999},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.30499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":657404,"kb_used_data":347572,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93714436,"statfs":{"total":96636764160,"available":95963582464,"internally_reserved":0,"allocated":355913728,"data_stored":355749689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.224}]},{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.282,"15min":1.1299999999999999},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":7.0899999999999999,"5min":19.343,"15min":19.343},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.373,"15min":1.1830000000000001},"min":{"1min":0.248,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5819999999999999,"5min":23.311,"15min":23.311},"last":0.38800000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.26200000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567540,"kb_used_data":257196,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93804300,"statfs":{"total":96636764160,"available":96055603200,"internally_reserved":0,"allocated":263368704,"data_stored":263201519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.17799999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.30599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":0.40000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":1.1639999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654116,"kb_used_data":347996,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93717724,"statfs":{"total":96636764160,"available":95966949376,"internally_reserved":0,"allocated":356347904,"data_stored":356180719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.173,"15min":0.89300000000000002},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.4100000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":6.4989999999999997},{"interface":"front","average":{"1min":3.6070000000000002,"5min":1.9330000000000001,"15min":1.657},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":27.736999999999998,"5min":27.736999999999998,"15min":27.736999999999998},"last":6.4749999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.51900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.74299999999999999},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.77300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355438592,"data_stored":355438592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263036928,"data_stored":263036928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354807808,"data_stored":354807808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355094528,"data_stored":355094528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:18.051 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:19.109 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:19.165 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:19.178 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T03:10:18.769097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":700938836,"num_objects":85569,"num_object_clones":0,"num_object_copies":342276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105138,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2520,"num_bytes_recovered":19972096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105165,"ondisk_log_size":105165,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2486084,"kb_used_data":1286860,"kb_used_omap":14,"kb_used_meta":1199153,"kb_avail":375001276,"statfs":{"total":386547056640,"available":384001306624,"internally_reserved":0,"allocated":1317744640,"data_stored":1317079046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227932699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,8,4,8,2,3,0,0,0,0,2,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28712960,"num_objects":-3505,"num_object_clones":0,"num_object_copies":-14020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002981"},"pg_stats":[{"pgid":"1.f","version":"33'6544","reported_seq":6593,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.681501+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:17.681501+0000","last_peered":"2024-09-15T03:10:17.681501+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:17.681501+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43958272,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6544,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6306","reported_seq":6376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.205797+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:17.205797+0000","last_peered":"2024-09-15T03:10:17.205797+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:17.205797+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41940594,"num_objects":5120,"num_object_clones":0,"num_object_copies":20480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6304,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6748","reported_seq":6842,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.833778+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:17.833778+0000","last_peered":"2024-09-15T03:10:17.833778+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:17.833778+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44825425,"num_objects":5473,"num_object_clones":0,"num_object_copies":21892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6747,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6397","reported_seq":7367,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.298570+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:17.298570+0000","last_peered":"2024-09-15T03:10:17.298570+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:17.298570+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6397,"log_dups_size":0,"ondisk_log_size":6397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42849906,"num_objects":5231,"num_object_clones":0,"num_object_copies":20924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6395,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":3653632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6678","reported_seq":6746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.575383+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:17.575383+0000","last_peered":"2024-09-15T03:10:17.575383+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:17.575383+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6678,"log_dups_size":0,"ondisk_log_size":6678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44258930,"num_objects":5403,"num_object_clones":0,"num_object_copies":21612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6676,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6639","reported_seq":6695,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.561717+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:17.561717+0000","last_peered":"2024-09-15T03:10:17.561717+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:17.561717+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43900928,"num_objects":5359,"num_object_clones":0,"num_object_copies":21436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6639,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6609","reported_seq":6684,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.309901+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:17.309901+0000","last_peered":"2024-09-15T03:10:17.309901+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:17.309901+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6609,"log_dups_size":0,"ondisk_log_size":6609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43816562,"num_objects":5349,"num_object_clones":0,"num_object_copies":21396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6607,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3301}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6677","reported_seq":6752,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.247903+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:17.247903+0000","last_peered":"2024-09-15T03:10:17.247903+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:17.247903+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44299890,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6614","reported_seq":6685,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.665764+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:17.665764+0000","last_peered":"2024-09-15T03:10:17.665764+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:17.665764+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6614,"log_dups_size":0,"ondisk_log_size":6614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43871460,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6611,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6611","reported_seq":6696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.944016+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:17.944016+0000","last_peered":"2024-09-15T03:10:17.944016+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:17.944016+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43882098,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3299}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6563","reported_seq":6627,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.548766+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:17.548766+0000","last_peered":"2024-09-15T03:10:17.548766+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:17.548766+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43577771,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6522","reported_seq":10561,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:17.954361+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:17.954361+0000","last_peered":"2024-09-15T03:10:17.954361+0000","last_clean":"2024-09-15T03:10:17.954361+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:17.954361+0000","last_undegraded":"2024-09-15T03:10:17.954361+0000","last_fullsized":"2024-09-15T03:10:17.954361+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44047161,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6521,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6672","reported_seq":6753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.899830+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:17.899830+0000","last_peered":"2024-09-15T03:10:17.899830+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:17.899830+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44735289,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6421","reported_seq":6479,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.423829+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:17.423829+0000","last_peered":"2024-09-15T03:10:17.423829+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:17.423829+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6421,"log_dups_size":0,"ondisk_log_size":6421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095666,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6419,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3228}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6532","reported_seq":6617,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.856114+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:17.856114+0000","last_peered":"2024-09-15T03:10:17.856114+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:17.856114+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43832946,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6632","reported_seq":6707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.098151+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:17.098151+0000","last_peered":"2024-09-15T03:10:17.098151+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:17.098151+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44045938,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6630,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":700938836,"num_objects":85569,"num_object_clones":0,"num_object_copies":342276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105138,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2520,"num_bytes_recovered":19972096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314009088,"data_stored":1314009088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105165,"ondisk_log_size":105165,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649556,"kb_used_data":343372,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93722284,"statfs":{"total":96636764160,"available":95971618816,"internally_reserved":0,"allocated":351612928,"data_stored":351445743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,1,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.19800000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.53800000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640440,"kb_used_data":343080,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93731400,"statfs":{"total":96636764160,"available":95980953600,"internally_reserved":0,"allocated":351313920,"data_stored":351149881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":5.4359999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.221}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.45100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.47199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554864,"kb_used_data":257000,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93816976,"statfs":{"total":96636764160,"available":96068583424,"internally_reserved":0,"allocated":263168000,"data_stored":263000815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":12.244}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":3.3079999999999998},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":5.5250000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.4990000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.42399999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641224,"kb_used_data":343408,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93730616,"statfs":{"total":96636764160,"available":95980150784,"internally_reserved":0,"allocated":351649792,"data_stored":351482607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2,4,0,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":2.105},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":2.1190000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":2.395}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":2.4300000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":2.5030000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350724096,"data_stored":350724096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262860800,"data_stored":262860800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":350068736,"data_stored":350068736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":350355456,"data_stored":350355456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:19.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:20.504 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:20.548 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:20.560 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T03:10:18.769097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":700938836,"num_objects":85569,"num_object_clones":0,"num_object_copies":342276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105138,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2520,"num_bytes_recovered":19972096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105165,"ondisk_log_size":105165,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2486084,"kb_used_data":1286860,"kb_used_omap":14,"kb_used_meta":1199153,"kb_avail":375001276,"statfs":{"total":386547056640,"available":384001306624,"internally_reserved":0,"allocated":1317744640,"data_stored":1317079046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1227932699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,8,4,8,2,3,0,0,0,0,2,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28712960,"num_objects":-3505,"num_object_clones":0,"num_object_copies":-14020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3505,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1581056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002981"},"pg_stats":[{"pgid":"1.f","version":"33'6544","reported_seq":6593,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.681501+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:17.681501+0000","last_peered":"2024-09-15T03:10:17.681501+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:17.681501+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6544,"log_dups_size":0,"ondisk_log_size":6544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43958272,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6544,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6306","reported_seq":6376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.205797+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:17.205797+0000","last_peered":"2024-09-15T03:10:17.205797+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:17.205797+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41940594,"num_objects":5120,"num_object_clones":0,"num_object_copies":20480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6304,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6748","reported_seq":6842,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.833778+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:17.833778+0000","last_peered":"2024-09-15T03:10:17.833778+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:17.833778+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44825425,"num_objects":5473,"num_object_clones":0,"num_object_copies":21892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6747,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6397","reported_seq":7367,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.298570+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:17.298570+0000","last_peered":"2024-09-15T03:10:17.298570+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:17.298570+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6397,"log_dups_size":0,"ondisk_log_size":6397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42849906,"num_objects":5231,"num_object_clones":0,"num_object_copies":20924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6395,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":3653632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6678","reported_seq":6746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.575383+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:17.575383+0000","last_peered":"2024-09-15T03:10:17.575383+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:17.575383+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6678,"log_dups_size":0,"ondisk_log_size":6678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44258930,"num_objects":5403,"num_object_clones":0,"num_object_copies":21612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6676,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3301}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6639","reported_seq":6695,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.561717+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:17.561717+0000","last_peered":"2024-09-15T03:10:17.561717+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:17.561717+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43900928,"num_objects":5359,"num_object_clones":0,"num_object_copies":21436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6639,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3308}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6609","reported_seq":6684,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.309901+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:17.309901+0000","last_peered":"2024-09-15T03:10:17.309901+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:17.309901+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6609,"log_dups_size":0,"ondisk_log_size":6609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43816562,"num_objects":5349,"num_object_clones":0,"num_object_copies":21396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6607,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3301}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6677","reported_seq":6752,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.247903+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:17.247903+0000","last_peered":"2024-09-15T03:10:17.247903+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:17.247903+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44299890,"num_objects":5408,"num_object_clones":0,"num_object_copies":21632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6675,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6614","reported_seq":6685,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.665764+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:17.665764+0000","last_peered":"2024-09-15T03:10:17.665764+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:17.665764+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6614,"log_dups_size":0,"ondisk_log_size":6614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43871460,"num_objects":5356,"num_object_clones":0,"num_object_copies":21424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6611,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6611","reported_seq":6696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.944016+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:17.944016+0000","last_peered":"2024-09-15T03:10:17.944016+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:17.944016+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43882098,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3299}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6563","reported_seq":6627,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.548766+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:17.548766+0000","last_peered":"2024-09-15T03:10:17.548766+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:17.548766+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43577771,"num_objects":5320,"num_object_clones":0,"num_object_copies":21280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6522","reported_seq":10561,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:17.954361+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:17.954361+0000","last_peered":"2024-09-15T03:10:17.954361+0000","last_clean":"2024-09-15T03:10:17.954361+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:17.954361+0000","last_undegraded":"2024-09-15T03:10:17.954361+0000","last_fullsized":"2024-09-15T03:10:17.954361+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44047161,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6521,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6672","reported_seq":6753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.899830+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:17.899830+0000","last_peered":"2024-09-15T03:10:17.899830+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:17.899830+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44735289,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6421","reported_seq":6479,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.423829+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:17.423829+0000","last_peered":"2024-09-15T03:10:17.423829+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:17.423829+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6421,"log_dups_size":0,"ondisk_log_size":6421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095666,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6419,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3228}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6532","reported_seq":6617,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.856114+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:17.856114+0000","last_peered":"2024-09-15T03:10:17.856114+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:17.856114+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43832946,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6632","reported_seq":6707,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.098151+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:17.098151+0000","last_peered":"2024-09-15T03:10:17.098151+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:17.098151+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44045938,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6630,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":700938836,"num_objects":85569,"num_object_clones":0,"num_object_copies":342276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105138,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2520,"num_bytes_recovered":19972096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314009088,"data_stored":1314009088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105165,"ondisk_log_size":105165,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649556,"kb_used_data":343372,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93722284,"statfs":{"total":96636764160,"available":95971618816,"internally_reserved":0,"allocated":351612928,"data_stored":351445743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,1,1,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.19800000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.53800000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640440,"kb_used_data":343080,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93731400,"statfs":{"total":96636764160,"available":95980953600,"internally_reserved":0,"allocated":351313920,"data_stored":351149881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":5.4359999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.221}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.45100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.47199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554864,"kb_used_data":257000,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93816976,"statfs":{"total":96636764160,"available":96068583424,"internally_reserved":0,"allocated":263168000,"data_stored":263000815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":12.244}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":3.3079999999999998},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":5.5250000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.4990000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.42399999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":641224,"kb_used_data":343408,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93730616,"statfs":{"total":96636764160,"available":95980150784,"internally_reserved":0,"allocated":351649792,"data_stored":351482607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2,4,0,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":2.105},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":2.1190000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":2.395}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":2.4300000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":2.5030000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350724096,"data_stored":350724096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262860800,"data_stored":262860800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":350068736,"data_stored":350068736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":350355456,"data_stored":350355456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:20.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:10:20.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6544", 'reported_seq': 6593, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.681501+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:10:17.681501+0000', 'last_peered': '2024-09-15T03:10:17.681501+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:10:17.681501+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6544, 'log_dups_size': 0, 'ondisk_log_size': 6544, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43958272, 'num_objects': 5366, 'num_object_clones': 0, 'num_object_copies': 21464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1973, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6544, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3393}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:20.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:10:20.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6306", 'reported_seq': 6376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.205797+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:10:17.205797+0000', 'last_peered': '2024-09-15T03:10:17.205797+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:10:17.205797+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6306, 'log_dups_size': 0, 'ondisk_log_size': 6306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41940594, 'num_objects': 5120, 'num_object_clones': 0, 'num_object_copies': 20480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6304, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:20.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:10:20.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6748", 'reported_seq': 6842, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.833778+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:10:17.833778+0000', 'last_peered': '2024-09-15T03:10:17.833778+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:10:17.833778+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6748, 'log_dups_size': 0, 'ondisk_log_size': 6748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44825425, 'num_objects': 5473, 'num_object_clones': 0, 'num_object_copies': 21892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5473, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6747, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3376}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:20.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6397", 'reported_seq': 7367, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.298570+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:10:17.298570+0000', 'last_peered': '2024-09-15T03:10:17.298570+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:10:17.298570+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6397, 'log_dups_size': 0, 'ondisk_log_size': 6397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42849906, 'num_objects': 5231, 'num_object_clones': 0, 'num_object_copies': 20924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1491, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5231, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6395, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 454, 'num_bytes_recovered': 3653632, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3740}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6678", 'reported_seq': 6746, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.575383+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:10:17.575383+0000', 'last_peered': '2024-09-15T03:10:17.575383+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:10:17.575383+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6678, 'log_dups_size': 0, 'ondisk_log_size': 6678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44258930, 'num_objects': 5403, 'num_object_clones': 0, 'num_object_copies': 21612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6676, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3301}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6639", 'reported_seq': 6695, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.561717+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:10:17.561717+0000', 'last_peered': '2024-09-15T03:10:17.561717+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:10:17.561717+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6639, 'log_dups_size': 0, 'ondisk_log_size': 6639, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43900928, 'num_objects': 5359, 'num_object_clones': 0, 'num_object_copies': 21436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6639, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3308}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6609", 'reported_seq': 6684, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.309901+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:10:17.309901+0000', 'last_peered': '2024-09-15T03:10:17.309901+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:10:17.309901+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6609, 'log_dups_size': 0, 'ondisk_log_size': 6609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43816562, 'num_objects': 5349, 'num_object_clones': 0, 'num_object_copies': 21396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6607, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3301}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6677", 'reported_seq': 6752, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.247903+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:10:17.247903+0000', 'last_peered': '2024-09-15T03:10:17.247903+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:10:17.247903+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6677, 'log_dups_size': 0, 'ondisk_log_size': 6677, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44299890, 'num_objects': 5408, 'num_object_clones': 0, 'num_object_copies': 21632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2078, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6675, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:20.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6614", 'reported_seq': 6685, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.665764+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:10:17.665764+0000', 'last_peered': '2024-09-15T03:10:17.665764+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:10:17.665764+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6614, 'log_dups_size': 0, 'ondisk_log_size': 6614, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43871460, 'num_objects': 5356, 'num_object_clones': 0, 'num_object_copies': 21424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5356, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6611, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3320}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6611", 'reported_seq': 6696, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.944016+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:10:17.944016+0000', 'last_peered': '2024-09-15T03:10:17.944016+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:10:17.944016+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6611, 'log_dups_size': 0, 'ondisk_log_size': 6611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43882098, 'num_objects': 5357, 'num_object_clones': 0, 'num_object_copies': 21428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6609, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3299}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6563", 'reported_seq': 6627, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.548766+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:10:17.548766+0000', 'last_peered': '2024-09-15T03:10:17.548766+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:10:17.548766+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6563, 'log_dups_size': 0, 'ondisk_log_size': 6563, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43577771, 'num_objects': 5320, 'num_object_clones': 0, 'num_object_copies': 21280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6560, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3244}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6672", 'reported_seq': 6753, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.899830+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:10:17.899830+0000', 'last_peered': '2024-09-15T03:10:17.899830+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:10:17.899830+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6672, 'log_dups_size': 0, 'ondisk_log_size': 6672, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44735289, 'num_objects': 5461, 'num_object_clones': 0, 'num_object_copies': 21844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6671, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6421", 'reported_seq': 6479, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.423829+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:10:17.423829+0000', 'last_peered': '2024-09-15T03:10:17.423829+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:10:17.423829+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6421, 'log_dups_size': 0, 'ondisk_log_size': 6421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43095666, 'num_objects': 5261, 'num_object_clones': 0, 'num_object_copies': 21044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2035, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6419, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3228}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:20.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:10:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6532", 'reported_seq': 6617, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.856114+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:10:17.856114+0000', 'last_peered': '2024-09-15T03:10:17.856114+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:10:17.856114+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6532, 'log_dups_size': 0, 'ondisk_log_size': 6532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43832946, 'num_objects': 5351, 'num_object_clones': 0, 'num_object_copies': 21404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6530, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:10:20.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6632", 'reported_seq': 6707, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:17.098151+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:10:17.098151+0000', 'last_peered': '2024-09-15T03:10:17.098151+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:10:17.098151+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6632, 'log_dups_size': 0, 'ondisk_log_size': 6632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44045938, 'num_objects': 5377, 'num_object_clones': 0, 'num_object_copies': 21508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6630, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3300}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:20.566 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:10:22.072 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:22.072 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30162/342276 objects degraded (8.812%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":345,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85569,"data_bytes":700938836,"bytes_used":2545750016,"bytes_avail":384001306624,"bytes_total":386547056640,"degraded_objects":30162,"degraded_total":342276,"degraded_ratio":0.088121866563825688,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":96231,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":1155072,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":192},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:10:22.100 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:10:22.100 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:23.908 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:23.937 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:23.962 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-15T03:10:22.770091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":694622804,"num_objects":84798,"num_object_clones":0,"num_object_copies":339192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84798,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105909,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2626,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105936,"ondisk_log_size":105936,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2472100,"kb_used_data":1277044,"kb_used_omap":14,"kb_used_meta":1194993,"kb_avail":375015260,"statfs":{"total":386547056640,"available":384015626240,"internally_reserved":0,"allocated":1307693056,"data_stored":1307027462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1223672859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,6,4,14,6,5,1,0,0,0,0,0,3],"upper_bound":32768},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25206784,"num_objects":-3077,"num_object_clones":0,"num_object_copies":-12308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3077,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":257,"num_bytes_recovered":1826816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002990"},"pg_stats":[{"pgid":"1.f","version":"33'6619","reported_seq":6675,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.711123+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:22.711123+0000","last_peered":"2024-09-15T03:10:22.711123+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:22.711123+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43343872,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3321}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6371","reported_seq":6443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.401850+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:22.401850+0000","last_peered":"2024-09-15T03:10:22.401850+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:22.401850+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41408114,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6369,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6748","reported_seq":6842,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.833778+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:17.833778+0000","last_peered":"2024-09-15T03:10:17.833778+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:17.833778+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44825425,"num_objects":5473,"num_object_clones":0,"num_object_copies":21892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6747,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6445","reported_seq":7578,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.376506+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:22.376506+0000","last_peered":"2024-09-15T03:10:22.376506+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:22.376506+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42456690,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":4325376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6758","reported_seq":6835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.737111+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:22.737111+0000","last_peered":"2024-09-15T03:10:22.737111+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:22.737111+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6758,"log_dups_size":0,"ondisk_log_size":6758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43603570,"num_objects":5323,"num_object_clones":0,"num_object_copies":21292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6712","reported_seq":6772,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.682195+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:22.682195+0000","last_peered":"2024-09-15T03:10:22.682195+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:22.682195+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6712,"log_dups_size":0,"ondisk_log_size":6712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43302912,"num_objects":5286,"num_object_clones":0,"num_object_copies":21144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3237}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6671","reported_seq":6750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:21.885017+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:21.885017+0000","last_peered":"2024-09-15T03:10:21.885017+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:21.885017+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6671,"log_dups_size":0,"ondisk_log_size":6671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43308658,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6669,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3241}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6740","reported_seq":6817,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.117647+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:22.117647+0000","last_peered":"2024-09-15T03:10:22.117647+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:22.117647+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43783794,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6690","reported_seq":6765,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.646958+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:22.646958+0000","last_peered":"2024-09-15T03:10:22.646958+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:22.646958+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6690,"log_dups_size":0,"ondisk_log_size":6690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43248868,"num_objects":5280,"num_object_clones":0,"num_object_copies":21120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6611","reported_seq":6696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.944016+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:17.944016+0000","last_peered":"2024-09-15T03:10:17.944016+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:17.944016+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43882098,"num_objects":5357,"num_object_clones":0,"num_object_copies":21428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6609,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3299}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6636","reported_seq":6704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.626374+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:22.626374+0000","last_peered":"2024-09-15T03:10:22.626374+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:22.626374+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6636,"log_dups_size":0,"ondisk_log_size":6636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42979755,"num_objects":5247,"num_object_clones":0,"num_object_copies":20988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6522","reported_seq":10561,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:17.954361+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:17.954361+0000","last_peered":"2024-09-15T03:10:17.954361+0000","last_clean":"2024-09-15T03:10:17.954361+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:17.954361+0000","last_undegraded":"2024-09-15T03:10:17.954361+0000","last_fullsized":"2024-09-15T03:10:17.954361+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44047161,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6521,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6672","reported_seq":6753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.899830+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:17.899830+0000","last_peered":"2024-09-15T03:10:17.899830+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:17.899830+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6672,"log_dups_size":0,"ondisk_log_size":6672,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44735289,"num_objects":5461,"num_object_clones":0,"num_object_copies":21844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6493","reported_seq":6563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.641565+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:22.641565+0000","last_peered":"2024-09-15T03:10:22.641565+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:22.641565+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6493,"log_dups_size":0,"ondisk_log_size":6493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42505842,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6532","reported_seq":6617,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:17.856114+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:17.856114+0000","last_peered":"2024-09-15T03:10:17.856114+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:17.856114+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43832946,"num_objects":5351,"num_object_clones":0,"num_object_copies":21404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6530,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6716","reported_seq":6791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.333876+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:22.333876+0000","last_peered":"2024-09-15T03:10:22.333876+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:22.333876+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43357810,"num_objects":5293,"num_object_clones":0,"num_object_copies":21172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":694622804,"num_objects":84798,"num_object_clones":0,"num_object_copies":339192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84798,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105909,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2626,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1304760320,"data_stored":1304760320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105936,"ondisk_log_size":105936,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635816,"kb_used_data":337948,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93736024,"statfs":{"total":96636764160,"available":95985688576,"internally_reserved":0,"allocated":346058752,"data_stored":345891567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,1,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.216}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.317}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.45500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640440,"kb_used_data":343080,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93731400,"statfs":{"total":96636764160,"available":95980953600,"internally_reserved":0,"allocated":351313920,"data_stored":351149881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":5.4359999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":0.221}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.45100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":0.47199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554864,"kb_used_data":257000,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93816976,"statfs":{"total":96636764160,"available":96068583424,"internally_reserved":0,"allocated":263168000,"data_stored":263000815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":12.244}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":3.3079999999999998},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":5.5250000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.4990000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.42399999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640980,"kb_used_data":339016,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93730860,"statfs":{"total":96636764160,"available":95980400640,"internally_reserved":0,"allocated":347152384,"data_stored":346985199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,7,1,3,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.0680000000000001},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":3.032}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346095616,"data_stored":346095616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262860800,"data_stored":262860800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":350068736,"data_stored":350068736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345735168,"data_stored":345735168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:26.964 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:28.199 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:28.211 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:28.225 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T03:10:26.772898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":691632724,"num_objects":84433,"num_object_clones":0,"num_object_copies":337732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84433,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106274,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106301,"ondisk_log_size":106301,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2479992,"kb_used_data":1272460,"kb_used_omap":14,"kb_used_meta":1207473,"kb_avail":375007368,"statfs":{"total":386547056640,"available":384007544832,"internally_reserved":0,"allocated":1302999040,"data_stored":1302333446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236452379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,10,7,8,20,10,9,1,0,1,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18694144,"num_objects":-2282,"num_object_clones":0,"num_object_copies":-9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1228800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004788"},"pg_stats":[{"pgid":"1.f","version":"33'6619","reported_seq":6675,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.711123+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:22.711123+0000","last_peered":"2024-09-15T03:10:22.711123+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:22.711123+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6619,"log_dups_size":0,"ondisk_log_size":6619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43343872,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6619,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3321}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6371","reported_seq":6443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.401850+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:22.401850+0000","last_peered":"2024-09-15T03:10:22.401850+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:22.401850+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41408114,"num_objects":5055,"num_object_clones":0,"num_object_copies":20220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6369,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6818","reported_seq":6916,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.767891+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:22.767891+0000","last_peered":"2024-09-15T03:10:22.767891+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:22.767891+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44251985,"num_objects":5403,"num_object_clones":0,"num_object_copies":21612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6817,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3308}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6445","reported_seq":7578,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.376506+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:22.376506+0000","last_peered":"2024-09-15T03:10:22.376506+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:22.376506+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6445,"log_dups_size":0,"ondisk_log_size":6445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42456690,"num_objects":5183,"num_object_clones":0,"num_object_copies":20732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1409,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":4325376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6758","reported_seq":6835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.737111+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:22.737111+0000","last_peered":"2024-09-15T03:10:22.737111+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:22.737111+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6758,"log_dups_size":0,"ondisk_log_size":6758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43603570,"num_objects":5323,"num_object_clones":0,"num_object_copies":21292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3226}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6712","reported_seq":6772,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.682195+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:22.682195+0000","last_peered":"2024-09-15T03:10:22.682195+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:22.682195+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6712,"log_dups_size":0,"ondisk_log_size":6712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43302912,"num_objects":5286,"num_object_clones":0,"num_object_copies":21144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3237}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6671","reported_seq":6750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:21.885017+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:21.885017+0000","last_peered":"2024-09-15T03:10:21.885017+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:21.885017+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6671,"log_dups_size":0,"ondisk_log_size":6671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43308658,"num_objects":5287,"num_object_clones":0,"num_object_copies":21148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6669,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3241}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6740","reported_seq":6817,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.117647+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:22.117647+0000","last_peered":"2024-09-15T03:10:22.117647+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:22.117647+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6740,"log_dups_size":0,"ondisk_log_size":6740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43783794,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6738,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6690","reported_seq":6765,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.646958+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:22.646958+0000","last_peered":"2024-09-15T03:10:22.646958+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:22.646958+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6690,"log_dups_size":0,"ondisk_log_size":6690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43248868,"num_objects":5280,"num_object_clones":0,"num_object_copies":21120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6686","reported_seq":6779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.873648+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:22.873648+0000","last_peered":"2024-09-15T03:10:22.873648+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:22.873648+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6686,"log_dups_size":0,"ondisk_log_size":6686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43267698,"num_objects":5282,"num_object_clones":0,"num_object_copies":21128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6684,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6636","reported_seq":6704,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.626374+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:22.626374+0000","last_peered":"2024-09-15T03:10:22.626374+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:22.626374+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6636,"log_dups_size":0,"ondisk_log_size":6636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42979755,"num_objects":5247,"num_object_clones":0,"num_object_copies":20988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6601","reported_seq":10640,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:23.028482+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:23.028482+0000","last_peered":"2024-09-15T03:10:23.028482+0000","last_clean":"2024-09-15T03:10:23.028482+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:23.028482+0000","last_undegraded":"2024-09-15T03:10:23.028482+0000","last_fullsized":"2024-09-15T03:10:23.028482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43399993,"num_objects":5298,"num_object_clones":0,"num_object_copies":21192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6600,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6746","reported_seq":6829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.821549+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:22.821549+0000","last_peered":"2024-09-15T03:10:22.821549+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:22.821549+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44129081,"num_objects":5387,"num_object_clones":0,"num_object_copies":21548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6493","reported_seq":6563,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.641565+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:22.641565+0000","last_peered":"2024-09-15T03:10:22.641565+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:22.641565+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6493,"log_dups_size":0,"ondisk_log_size":6493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42505842,"num_objects":5189,"num_object_clones":0,"num_object_copies":20756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6599","reported_seq":6686,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.789766+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:22.789766+0000","last_peered":"2024-09-15T03:10:22.789766+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:22.789766+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43284082,"num_objects":5284,"num_object_clones":0,"num_object_copies":21136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6597,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3298}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6716","reported_seq":6791,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:22.333876+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:22.333876+0000","last_peered":"2024-09-15T03:10:22.333876+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:22.333876+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43357810,"num_objects":5293,"num_object_clones":0,"num_object_copies":21172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":691632724,"num_objects":84433,"num_object_clones":0,"num_object_copies":337732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84433,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":106274,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":20643840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1300029440,"data_stored":1300029440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106301,"ondisk_log_size":106301,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635816,"kb_used_data":337948,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93736024,"statfs":{"total":96636764160,"available":95985688576,"internally_reserved":0,"allocated":346058752,"data_stored":345891567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,1,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.216}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.317}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.45500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640164,"kb_used_data":338648,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93731676,"statfs":{"total":96636764160,"available":95981236224,"internally_reserved":0,"allocated":346775552,"data_stored":346611513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,1,6,3,4,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.9330000000000001,"5min":1.7729999999999999,"15min":1.0720000000000001},"min":{"1min":0.22600000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.687999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":1.8979999999999999},{"interface":"front","average":{"1min":1.9750000000000001,"5min":1.397,"15min":1.0129999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.8860000000000001,"5min":19.172999999999998,"15min":19.172999999999998},"last":1.8200000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.36899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":1.6279999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563032,"kb_used_data":256848,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93808808,"statfs":{"total":96636764160,"available":96060219392,"internally_reserved":0,"allocated":263012352,"data_stored":262845167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.316},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.874}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":4.5389999999999997},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":0.34899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640980,"kb_used_data":339016,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93730860,"statfs":{"total":96636764160,"available":95980400640,"internally_reserved":0,"allocated":347152384,"data_stored":346985199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,7,1,3,1,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.0680000000000001},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":3.032}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.33700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346095616,"data_stored":346095616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262664192,"data_stored":262664192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":345534464,"data_stored":345534464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345735168,"data_stored":345735168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:28.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:29.767 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:29.826 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:29.837 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-15T03:10:28.773348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":681720404,"num_objects":83223,"num_object_clones":0,"num_object_copies":332892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107484,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2710,"num_bytes_recovered":21225472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107511,"ondisk_log_size":107511,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2494360,"kb_used_data":1257668,"kb_used_omap":14,"kb_used_meta":1236593,"kb_avail":374993000,"statfs":{"total":386547056640,"available":383992832000,"internally_reserved":0,"allocated":1287852032,"data_stored":1287186438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266271259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,4,13,10,5,4,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28606464,"num_objects":-3492,"num_object_clones":0,"num_object_copies":-13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3492,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":1810432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004707"},"pg_stats":[{"pgid":"1.f","version":"33'6692","reported_seq":6748,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.716142+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:27.716142+0000","last_peered":"2024-09-15T03:10:27.716142+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:27.716142+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6692,"log_dups_size":0,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42745856,"num_objects":5218,"num_object_clones":0,"num_object_copies":20872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6444","reported_seq":6516,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.401719+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:27.401719+0000","last_peered":"2024-09-15T03:10:27.401719+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:27.401719+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40810098,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6442,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2988}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6897","reported_seq":6995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.928477+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:27.928477+0000","last_peered":"2024-09-15T03:10:27.928477+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:27.928477+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43604817,"num_objects":5324,"num_object_clones":0,"num_object_copies":21296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6896,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3229}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6500","reported_seq":7776,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.402880+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:27.402880+0000","last_peered":"2024-09-15T03:10:27.402880+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:27.402880+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6500,"log_dups_size":0,"ondisk_log_size":6500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42006130,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6498,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":607,"num_bytes_recovered":4907008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6829","reported_seq":6907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.770939+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:27.770939+0000","last_peered":"2024-09-15T03:10:27.770939+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:27.770939+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6829,"log_dups_size":0,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43021938,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6784","reported_seq":6844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.604554+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:27.604554+0000","last_peered":"2024-09-15T03:10:27.604554+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:27.604554+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6784,"log_dups_size":0,"ondisk_log_size":6784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42713088,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6742","reported_seq":6821,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.210937+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:27.210937+0000","last_peered":"2024-09-15T03:10:27.210937+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:27.210937+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6742,"log_dups_size":0,"ondisk_log_size":6742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42727026,"num_objects":5216,"num_object_clones":0,"num_object_copies":20864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6740,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3170}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6813","reported_seq":6890,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.352112+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:27.352112+0000","last_peered":"2024-09-15T03:10:27.352112+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:27.352112+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6813,"log_dups_size":0,"ondisk_log_size":6813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43185778,"num_objects":5272,"num_object_clones":0,"num_object_copies":21088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6811,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3197}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6767","reported_seq":6842,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.454639+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:27.454639+0000","last_peered":"2024-09-15T03:10:27.454639+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:27.454639+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6767,"log_dups_size":0,"ondisk_log_size":6767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42618084,"num_objects":5203,"num_object_clones":0,"num_object_copies":20812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6764,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6772","reported_seq":6865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.943963+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:27.943963+0000","last_peered":"2024-09-15T03:10:27.943963+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:27.943963+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6772,"log_dups_size":0,"ondisk_log_size":6772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42563186,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6770,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6711","reported_seq":6779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.499551+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:27.499551+0000","last_peered":"2024-09-15T03:10:27.499551+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:27.499551+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42365355,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6682","reported_seq":10721,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:27.990550+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:27.990550+0000","last_peered":"2024-09-15T03:10:27.990550+0000","last_clean":"2024-09-15T03:10:27.990550+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:27.990550+0000","last_undegraded":"2024-09-15T03:10:27.990550+0000","last_fullsized":"2024-09-15T03:10:27.990550+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42736441,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6681,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6818","reported_seq":6907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.854549+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:27.854549+0000","last_peered":"2024-09-15T03:10:27.854549+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:27.854549+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43539257,"num_objects":5315,"num_object_clones":0,"num_object_copies":21260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6817,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6578","reported_seq":6648,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.751459+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:27.751459+0000","last_peered":"2024-09-15T03:10:27.751459+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:27.751459+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6578,"log_dups_size":0,"ondisk_log_size":6578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41809522,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6576,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3077}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6675","reported_seq":6764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.916967+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:27.916967+0000","last_peered":"2024-09-15T03:10:27.916967+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:27.916967+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6675,"log_dups_size":0,"ondisk_log_size":6675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42661490,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6673,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3223}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6807","reported_seq":6882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.336408+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:27.336408+0000","last_peered":"2024-09-15T03:10:27.336408+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:27.336408+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6807,"log_dups_size":0,"ondisk_log_size":6807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42612338,"num_objects":5202,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3125}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":681720404,"num_objects":83223,"num_object_clones":0,"num_object_copies":332892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107484,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2710,"num_bytes_recovered":21225472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1284857856,"data_stored":1284857856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107511,"ondisk_log_size":107511,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639372,"kb_used_data":333188,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93732468,"statfs":{"total":96636764160,"available":95982047232,"internally_reserved":0,"allocated":341184512,"data_stored":341017327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":2.927}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":2.347},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":2.3849999999999998},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.307}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643576,"kb_used_data":333712,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93728264,"statfs":{"total":96636764160,"available":95977742336,"internally_reserved":0,"allocated":341721088,"data_stored":341557049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2,2,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":17.152999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":12.531000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":12.898},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":12.930999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":12.692},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":12.566000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566916,"kb_used_data":256572,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93804924,"statfs":{"total":96636764160,"available":96056242176,"internally_reserved":0,"allocated":262729728,"data_stored":262562543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":9.0079999999999991}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":3.391},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":0.69999999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":5.9009999999999998},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":3.4319999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644496,"kb_used_data":334196,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93727344,"statfs":{"total":96636764160,"available":95976800256,"internally_reserved":0,"allocated":342216704,"data_stored":342049519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,3,2,3,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.575},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":4.0220000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":4.1920000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":1.0980000000000001},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":1.0609999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341106688,"data_stored":341106688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262381568,"data_stored":262381568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340557824,"data_stored":340557824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340811776,"data_stored":340811776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:29.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:10:29.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6692", 'reported_seq': 6748, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.716142+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:10:27.716142+0000', 'last_peered': '2024-09-15T03:10:27.716142+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:10:27.716142+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6692, 'log_dups_size': 0, 'ondisk_log_size': 6692, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42745856, 'num_objects': 5218, 'num_object_clones': 0, 'num_object_copies': 20872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6692, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3248}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:29.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:10:29.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6444", 'reported_seq': 6516, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.401719+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:10:27.401719+0000', 'last_peered': '2024-09-15T03:10:27.401719+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:10:27.401719+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6444, 'log_dups_size': 0, 'ondisk_log_size': 6444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40810098, 'num_objects': 4982, 'num_object_clones': 0, 'num_object_copies': 19928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1996, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6442, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2988}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:29.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:10:29.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6897", 'reported_seq': 6995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.928477+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:10:27.928477+0000', 'last_peered': '2024-09-15T03:10:27.928477+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:10:27.928477+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6897, 'log_dups_size': 0, 'ondisk_log_size': 6897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43604817, 'num_objects': 5324, 'num_object_clones': 0, 'num_object_copies': 21296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2096, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5324, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6896, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3229}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:29.839 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:10:29.839 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6500", 'reported_seq': 7776, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.402880+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:10:27.402880+0000', 'last_peered': '2024-09-15T03:10:27.402880+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:10:27.402880+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6500, 'log_dups_size': 0, 'ondisk_log_size': 6500, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42006130, 'num_objects': 5128, 'num_object_clones': 0, 'num_object_copies': 20512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1338, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6498, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 607, 'num_bytes_recovered': 4907008, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:29.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:10:29.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6829", 'reported_seq': 6907, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.770939+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:10:27.770939+0000', 'last_peered': '2024-09-15T03:10:27.770939+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:10:27.770939+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6829, 'log_dups_size': 0, 'ondisk_log_size': 6829, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43021938, 'num_objects': 5252, 'num_object_clones': 0, 'num_object_copies': 21008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2099, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6827, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3155}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:29.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:10:29.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6784", 'reported_seq': 6844, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.604554+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:10:27.604554+0000', 'last_peered': '2024-09-15T03:10:27.604554+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:10:27.604554+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6784, 'log_dups_size': 0, 'ondisk_log_size': 6784, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42713088, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6784, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3165}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:29.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:10:29.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6742", 'reported_seq': 6821, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.210937+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:10:27.210937+0000', 'last_peered': '2024-09-15T03:10:27.210937+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:10:27.210937+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6742, 'log_dups_size': 0, 'ondisk_log_size': 6742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42727026, 'num_objects': 5216, 'num_object_clones': 0, 'num_object_copies': 20864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2048, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6740, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3170}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:29.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:10:29.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6813", 'reported_seq': 6890, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.352112+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:10:27.352112+0000', 'last_peered': '2024-09-15T03:10:27.352112+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:10:27.352112+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6813, 'log_dups_size': 0, 'ondisk_log_size': 6813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43185778, 'num_objects': 5272, 'num_object_clones': 0, 'num_object_copies': 21088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2077, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6811, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3197}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:29.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:10:29.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6767", 'reported_seq': 6842, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.454639+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:10:27.454639+0000', 'last_peered': '2024-09-15T03:10:27.454639+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:10:27.454639+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6767, 'log_dups_size': 0, 'ondisk_log_size': 6767, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42618084, 'num_objects': 5203, 'num_object_clones': 0, 'num_object_copies': 20812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6764, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3169}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:29.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:10:29.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6772", 'reported_seq': 6865, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.943963+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:10:27.943963+0000', 'last_peered': '2024-09-15T03:10:27.943963+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:10:27.943963+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6772, 'log_dups_size': 0, 'ondisk_log_size': 6772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42563186, 'num_objects': 5196, 'num_object_clones': 0, 'num_object_copies': 20784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2055, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6770, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3143}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:29.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:10:29.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6711", 'reported_seq': 6779, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.499551+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:10:27.499551+0000', 'last_peered': '2024-09-15T03:10:27.499551+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:10:27.499551+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6711, 'log_dups_size': 0, 'ondisk_log_size': 6711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42365355, 'num_objects': 5172, 'num_object_clones': 0, 'num_object_copies': 20688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2077, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6708, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3098}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:29.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:10:29.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6818", 'reported_seq': 6907, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.854549+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:10:27.854549+0000', 'last_peered': '2024-09-15T03:10:27.854549+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:10:27.854549+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6818, 'log_dups_size': 0, 'ondisk_log_size': 6818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43539257, 'num_objects': 5315, 'num_object_clones': 0, 'num_object_copies': 21260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2035, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6817, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:29.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:10:29.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6578", 'reported_seq': 6648, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.751459+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:10:27.751459+0000', 'last_peered': '2024-09-15T03:10:27.751459+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:10:27.751459+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6578, 'log_dups_size': 0, 'ondisk_log_size': 6578, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41809522, 'num_objects': 5104, 'num_object_clones': 0, 'num_object_copies': 20416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2029, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6576, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3077}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:29.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:10:29.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6675", 'reported_seq': 6764, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.916967+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:10:27.916967+0000', 'last_peered': '2024-09-15T03:10:27.916967+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:10:27.916967+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6675, 'log_dups_size': 0, 'ondisk_log_size': 6675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42661490, 'num_objects': 5208, 'num_object_clones': 0, 'num_object_copies': 20832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1987, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6673, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3223}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:29.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:10:29.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6807", 'reported_seq': 6882, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:27.336408+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:10:27.336408+0000', 'last_peered': '2024-09-15T03:10:27.336408+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:10:27.336408+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6807, 'log_dups_size': 0, 'ondisk_log_size': 6807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42612338, 'num_objects': 5202, 'num_object_clones': 0, 'num_object_copies': 20808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2079, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6805, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3125}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:29.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:10:31.499 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:31.499 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29972/332892 objects degraded (9.004%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":354,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83223,"data_bytes":681720404,"bytes_used":2554224640,"bytes_avail":383992832000,"bytes_total":386547056640,"degraded_objects":29972,"degraded_total":332892,"degraded_ratio":0.090035206613556351,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":104384,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":1253376,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:10:31.563 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:10:31.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:33.024 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:33.078 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:33.100 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-15T03:10:32.778614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":678656596,"num_objects":82849,"num_object_clones":0,"num_object_copies":331396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107858,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2769,"num_bytes_recovered":21708800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107885,"ondisk_log_size":107885,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2477160,"kb_used_data":1252960,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375010200,"statfs":{"total":386547056640,"available":384010444800,"internally_reserved":0,"allocated":1283031040,"data_stored":1282365446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,6,13,16,4,3,1,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22282240,"num_objects":-2720,"num_object_clones":0,"num_object_copies":-10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008984"},"pg_stats":[{"pgid":"1.f","version":"33'6692","reported_seq":6748,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.716142+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:27.716142+0000","last_peered":"2024-09-15T03:10:27.716142+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:27.716142+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6692,"log_dups_size":0,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42745856,"num_objects":5218,"num_object_clones":0,"num_object_copies":20872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6508","reported_seq":6579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.348974+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:32.348974+0000","last_peered":"2024-09-15T03:10:32.348974+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:32.348974+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6508,"log_dups_size":0,"ondisk_log_size":6508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40285810,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6506,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6897","reported_seq":6995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.928477+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:27.928477+0000","last_peered":"2024-09-15T03:10:27.928477+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:27.928477+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43604817,"num_objects":5324,"num_object_clones":0,"num_object_copies":21296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5324,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6896,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3229}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6572","reported_seq":7966,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.467215+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:32.467215+0000","last_peered":"2024-09-15T03:10:32.467215+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:32.467215+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6572,"log_dups_size":0,"ondisk_log_size":6572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41416306,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6570,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":666,"num_bytes_recovered":5390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6829","reported_seq":6907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.770939+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:27.770939+0000","last_peered":"2024-09-15T03:10:27.770939+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:27.770939+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6829,"log_dups_size":0,"ondisk_log_size":6829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43021938,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3155}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6784","reported_seq":6844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.604554+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:27.604554+0000","last_peered":"2024-09-15T03:10:27.604554+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:27.604554+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6784,"log_dups_size":0,"ondisk_log_size":6784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42713088,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6822","reported_seq":6901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.433750+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:32.433750+0000","last_peered":"2024-09-15T03:10:32.433750+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:32.433750+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6822,"log_dups_size":0,"ondisk_log_size":6822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42071666,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6820,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3090}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6878","reported_seq":6955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.363116+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:32.363116+0000","last_peered":"2024-09-15T03:10:32.363116+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:32.363116+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42653298,"num_objects":5207,"num_object_clones":0,"num_object_copies":20828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6876,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6767","reported_seq":6842,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.454639+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:27.454639+0000","last_peered":"2024-09-15T03:10:27.454639+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:27.454639+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6767,"log_dups_size":0,"ondisk_log_size":6767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42618084,"num_objects":5203,"num_object_clones":0,"num_object_copies":20812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6764,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6772","reported_seq":6865,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.943963+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:27.943963+0000","last_peered":"2024-09-15T03:10:27.943963+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:27.943963+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6772,"log_dups_size":0,"ondisk_log_size":6772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42563186,"num_objects":5196,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6770,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3143}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6711","reported_seq":6779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.499551+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:27.499551+0000","last_peered":"2024-09-15T03:10:27.499551+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:27.499551+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6711,"log_dups_size":0,"ondisk_log_size":6711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42365355,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3098}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6682","reported_seq":10721,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:27.990550+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:27.990550+0000","last_peered":"2024-09-15T03:10:27.990550+0000","last_clean":"2024-09-15T03:10:27.990550+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:27.990550+0000","last_undegraded":"2024-09-15T03:10:27.990550+0000","last_fullsized":"2024-09-15T03:10:27.990550+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42736441,"num_objects":5217,"num_object_clones":0,"num_object_copies":20868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6681,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6818","reported_seq":6907,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.854549+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:27.854549+0000","last_peered":"2024-09-15T03:10:27.854549+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:27.854549+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6818,"log_dups_size":0,"ondisk_log_size":6818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43539257,"num_objects":5315,"num_object_clones":0,"num_object_copies":21260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6817,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6578","reported_seq":6648,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.751459+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:27.751459+0000","last_peered":"2024-09-15T03:10:27.751459+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:27.751459+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6578,"log_dups_size":0,"ondisk_log_size":6578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41809522,"num_objects":5104,"num_object_clones":0,"num_object_copies":20416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6576,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3077}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6675","reported_seq":6764,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:27.916967+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:27.916967+0000","last_peered":"2024-09-15T03:10:27.916967+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:27.916967+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6675,"log_dups_size":0,"ondisk_log_size":6675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42661490,"num_objects":5208,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6673,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3223}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6900","reported_seq":6975,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.369465+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:32.369465+0000","last_peered":"2024-09-15T03:10:32.369465+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:32.369465+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6900,"log_dups_size":0,"ondisk_log_size":6900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41850482,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":678656596,"num_objects":82849,"num_object_clones":0,"num_object_copies":331396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107858,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2769,"num_bytes_recovered":21708800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280057344,"data_stored":1280057344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107885,"ondisk_log_size":107885,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639372,"kb_used_data":333188,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93732468,"statfs":{"total":96636764160,"available":95982047232,"internally_reserved":0,"allocated":341184512,"data_stored":341017327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,3,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":2.927}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":2.347},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":2.3849999999999998},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.307}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643576,"kb_used_data":333712,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93728264,"statfs":{"total":96636764160,"available":95977742336,"internally_reserved":0,"allocated":341721088,"data_stored":341557049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2,2,0,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":17.152999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":12.531000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":12.898},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":12.930999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:09:30 2024","interfaces":[{"interface":"back","average":{"1min":2.0579999999999998,"5min":1.6930000000000001,"15min":0.96699999999999997},"min":{"1min":0.26800000000000002,"5min":0.216,"15min":0.216},"max":{"1min":15.675000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":12.692},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.552,"15min":1.03},"min":{"1min":0.27600000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":6.1210000000000004,"5min":19.274000000000001,"15min":19.274000000000001},"last":12.566000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566916,"kb_used_data":256572,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93804924,"statfs":{"total":96636764160,"available":96056242176,"internally_reserved":0,"allocated":262729728,"data_stored":262562543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":9.0079999999999991}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":3.391},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":0.69999999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":5.9009999999999998},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":3.4319999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627296,"kb_used_data":329488,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93744544,"statfs":{"total":96636764160,"available":95994413056,"internally_reserved":0,"allocated":337395712,"data_stored":337228527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,3,8,2,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":6.7830000000000004},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.6509999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.497},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":6.8710000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":6.8129999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336306176,"data_stored":336306176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262381568,"data_stored":262381568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":340557824,"data_stored":340557824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":340811776,"data_stored":340811776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:36.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:37.599 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:37.630 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:37.658 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-15T03:10:36.779640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":672086612,"num_objects":82047,"num_object_clones":0,"num_object_copies":328188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108660,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2769,"num_bytes_recovered":21708800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108687,"ondisk_log_size":108687,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2450844,"kb_used_data":1243316,"kb_used_omap":14,"kb_used_meta":1207473,"kb_avail":375036516,"statfs":{"total":386547056640,"available":384037392384,"internally_reserved":0,"allocated":1273155584,"data_stored":1272489990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236452379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,5,7,17,18,5,6,2,0,0,0,1,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19546112,"num_objects":-2386,"num_object_clones":0,"num_object_copies":-9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007229"},"pg_stats":[{"pgid":"1.f","version":"33'6756","reported_seq":6812,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.692787+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:32.692787+0000","last_peered":"2024-09-15T03:10:32.692787+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:32.692787+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6756,"log_dups_size":0,"ondisk_log_size":6756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42221568,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6756,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6508","reported_seq":6579,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.348974+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:32.348974+0000","last_peered":"2024-09-15T03:10:32.348974+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:32.348974+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6508,"log_dups_size":0,"ondisk_log_size":6508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40285810,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6506,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6974","reported_seq":7071,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:33.027194+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:33.027194+0000","last_peered":"2024-09-15T03:10:33.027194+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:33.027194+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6974,"log_dups_size":0,"ondisk_log_size":6974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42974033,"num_objects":5247,"num_object_clones":0,"num_object_copies":20988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6973,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3152}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6572","reported_seq":7966,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.467215+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:32.467215+0000","last_peered":"2024-09-15T03:10:32.467215+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:32.467215+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6572,"log_dups_size":0,"ondisk_log_size":6572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41416306,"num_objects":5056,"num_object_clones":0,"num_object_copies":20224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6570,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":666,"num_bytes_recovered":5390336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6905","reported_seq":6983,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.690115+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:32.690115+0000","last_peered":"2024-09-15T03:10:32.690115+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:32.690115+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6905,"log_dups_size":0,"ondisk_log_size":6905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42399346,"num_objects":5176,"num_object_clones":0,"num_object_copies":20704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6855","reported_seq":6915,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.799024+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:32.799024+0000","last_peered":"2024-09-15T03:10:32.799024+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:32.799024+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6855,"log_dups_size":0,"ondisk_log_size":6855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42131456,"num_objects":5143,"num_object_clones":0,"num_object_copies":20572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6855,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3094}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6822","reported_seq":6901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.433750+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:32.433750+0000","last_peered":"2024-09-15T03:10:32.433750+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:32.433750+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6822,"log_dups_size":0,"ondisk_log_size":6822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42071666,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6820,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3090}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6878","reported_seq":6955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.363116+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:32.363116+0000","last_peered":"2024-09-15T03:10:32.363116+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:32.363116+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6878,"log_dups_size":0,"ondisk_log_size":6878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42653298,"num_objects":5207,"num_object_clones":0,"num_object_copies":20828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6876,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3132}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6835","reported_seq":6910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.633490+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:32.633490+0000","last_peered":"2024-09-15T03:10:32.633490+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:32.633490+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42061028,"num_objects":5135,"num_object_clones":0,"num_object_copies":20540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6832,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6859","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:33.025524+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:33.025524+0000","last_peered":"2024-09-15T03:10:33.025524+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:33.025524+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6859,"log_dups_size":0,"ondisk_log_size":6859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41850482,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6857,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3056}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6768","reported_seq":6836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.731863+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:32.731863+0000","last_peered":"2024-09-15T03:10:32.731863+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:32.731863+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6768,"log_dups_size":0,"ondisk_log_size":6768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41898411,"num_objects":5115,"num_object_clones":0,"num_object_copies":20460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6765,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6763","reported_seq":10802,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:33.211818+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:33.211818+0000","last_peered":"2024-09-15T03:10:33.211818+0000","last_clean":"2024-09-15T03:10:33.211818+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:33.211818+0000","last_undegraded":"2024-09-15T03:10:33.211818+0000","last_fullsized":"2024-09-15T03:10:33.211818+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6763,"log_dups_size":0,"ondisk_log_size":6763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42072889,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6762,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6890","reported_seq":6979,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:33.002302+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:33.002302+0000","last_peered":"2024-09-15T03:10:33.002302+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:33.002302+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6890,"log_dups_size":0,"ondisk_log_size":6890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42949433,"num_objects":5243,"num_object_clones":0,"num_object_copies":20972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6889,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3209}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6655","reported_seq":6725,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.733724+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:32.733724+0000","last_peered":"2024-09-15T03:10:32.733724+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:32.733724+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6655,"log_dups_size":0,"ondisk_log_size":6655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41178738,"num_objects":5027,"num_object_clones":0,"num_object_copies":20108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6653,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3000}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6747","reported_seq":6836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.824167+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:32.824167+0000","last_peered":"2024-09-15T03:10:32.824167+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:32.824167+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6747,"log_dups_size":0,"ondisk_log_size":6747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42071666,"num_objects":5136,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3151}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6900","reported_seq":6975,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:32.369465+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:32.369465+0000","last_peered":"2024-09-15T03:10:32.369465+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:32.369465+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6900,"log_dups_size":0,"ondisk_log_size":6900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41850482,"num_objects":5109,"num_object_clones":0,"num_object_copies":20436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2079,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":672086612,"num_objects":82047,"num_object_clones":0,"num_object_copies":328188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29913,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82047,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108660,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2769,"num_bytes_recovered":21708800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1270104064,"data_stored":1270104064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108687,"ondisk_log_size":108687,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638912,"kb_used_data":328568,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93732928,"statfs":{"total":96636764160,"available":95982518272,"internally_reserved":0,"allocated":336453632,"data_stored":336286447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,10,7,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.192},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.972}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":10.185},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":2.3420000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":18.579999999999998},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.36099999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626416,"kb_used_data":329064,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93745424,"statfs":{"total":96636764160,"available":95995314176,"internally_reserved":0,"allocated":336961536,"data_stored":336797497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,2,2,0,2,1,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":1.714},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.66000000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558220,"kb_used_data":256196,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93813620,"statfs":{"total":96636764160,"available":96065146880,"internally_reserved":0,"allocated":262344704,"data_stored":262177519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.372},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":1.0469999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.307},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":0.39500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":3.1709999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":627296,"kb_used_data":329488,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93744544,"statfs":{"total":96636764160,"available":95994413056,"internally_reserved":0,"allocated":337395712,"data_stored":337228527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,3,8,2,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":6.7830000000000004},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.6509999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:09:31 2024","interfaces":[{"interface":"back","average":{"1min":2.8140000000000001,"5min":2.089,"15min":1.319},"min":{"1min":0.22900000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":18.808,"5min":30.802,"15min":30.802},"last":0.497},{"interface":"front","average":{"1min":1.819,"5min":1.161,"15min":0.79300000000000004},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":6.8710000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":6.8129999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336306176,"data_stored":336306176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262008832,"data_stored":262008832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335798272,"data_stored":335798272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":335990784,"data_stored":335990784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:37.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:39.163 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:39.204 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:39.233 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-15T03:10:38.780202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662501972,"num_objects":80877,"num_object_clones":0,"num_object_copies":323508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80877,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109830,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2830,"num_bytes_recovered":22151168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109857,"ondisk_log_size":109857,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2440364,"kb_used_data":1228672,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375046996,"statfs":{"total":386547056640,"available":384048123904,"internally_reserved":0,"allocated":1258160128,"data_stored":1257494534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,5,9,10,15,1,5,0,0,2,0,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29130752,"num_objects":-3556,"num_object_clones":0,"num_object_copies":-14224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3556,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":1507328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007303"},"pg_stats":[{"pgid":"1.f","version":"33'6827","reported_seq":6883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.826923+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:37.826923+0000","last_peered":"2024-09-15T03:10:37.826923+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:37.826923+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41639936,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6576","reported_seq":6650,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.430258+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:37.430258+0000","last_peered":"2024-09-15T03:10:37.430258+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:37.430258+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6576,"log_dups_size":0,"ondisk_log_size":6576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39728754,"num_objects":4850,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6574,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2857}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7044","reported_seq":7142,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.942258+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:37.942258+0000","last_peered":"2024-09-15T03:10:37.942258+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:37.942258+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42400593,"num_objects":5177,"num_object_clones":0,"num_object_copies":20708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7043,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3082}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6649","reported_seq":8149,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.442558+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:37.442558+0000","last_peered":"2024-09-15T03:10:37.442558+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:37.442558+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6649,"log_dups_size":0,"ondisk_log_size":6649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40785522,"num_objects":4979,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1225,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6647,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":720,"num_bytes_recovered":5832704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6983","reported_seq":7061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.799970+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:37.799970+0000","last_peered":"2024-09-15T03:10:37.799970+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:37.799970+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41760370,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6981,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6929","reported_seq":6989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.736215+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:37.736215+0000","last_peered":"2024-09-15T03:10:37.736215+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:37.736215+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6929,"log_dups_size":0,"ondisk_log_size":6929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41525248,"num_objects":5069,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6888","reported_seq":6967,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.348012+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:37.348012+0000","last_peered":"2024-09-15T03:10:37.348012+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:37.348012+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6888,"log_dups_size":0,"ondisk_log_size":6888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41530994,"num_objects":5070,"num_object_clones":0,"num_object_copies":20280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6886,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6951","reported_seq":7030,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.233649+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:37.233649+0000","last_peered":"2024-09-15T03:10:37.233649+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:37.233649+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6951,"log_dups_size":0,"ondisk_log_size":6951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42055282,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6949,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3060}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6906","reported_seq":6981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.841505+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:37.841505+0000","last_peered":"2024-09-15T03:10:37.841505+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:37.841505+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41479396,"num_objects":5064,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6945","reported_seq":7042,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:38.032788+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:38.032788+0000","last_peered":"2024-09-15T03:10:38.032788+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:38.032788+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41145970,"num_objects":5023,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6848","reported_seq":6914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.762472+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:37.762472+0000","last_peered":"2024-09-15T03:10:37.762472+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:37.762472+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243051,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6845,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6845","reported_seq":10884,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:38.141308+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:38.141308+0000","last_peered":"2024-09-15T03:10:38.141308+0000","last_clean":"2024-09-15T03:10:38.141308+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:38.141308+0000","last_undegraded":"2024-09-15T03:10:38.141308+0000","last_fullsized":"2024-09-15T03:10:38.141308+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41401145,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6844,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6963","reported_seq":7052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.660614+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:37.660614+0000","last_peered":"2024-09-15T03:10:37.660614+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:37.660614+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42351417,"num_objects":5170,"num_object_clones":0,"num_object_copies":20680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3136}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6718","reported_seq":6788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.746649+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:37.746649+0000","last_peered":"2024-09-15T03:10:37.746649+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:37.746649+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40662642,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6816","reported_seq":6905,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.847808+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:37.847808+0000","last_peered":"2024-09-15T03:10:37.847808+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:37.847808+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41506418,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6969","reported_seq":7052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.459056+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:37.459056+0000","last_peered":"2024-09-15T03:10:37.459056+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:37.459056+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6969,"log_dups_size":0,"ondisk_log_size":6969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41285234,"num_objects":5040,"num_object_clones":0,"num_object_copies":20160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6967,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2967}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":662501972,"num_objects":80877,"num_object_clones":0,"num_object_copies":323508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80877,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109830,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2830,"num_bytes_recovered":22151168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1255112704,"data_stored":1255112704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109857,"ondisk_log_size":109857,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621672,"kb_used_data":323796,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93750168,"statfs":{"total":96636764160,"available":96000172032,"internally_reserved":0,"allocated":331567104,"data_stored":331399919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,5,1,2,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.865},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.8460000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.307}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629948,"kb_used_data":324276,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93741892,"statfs":{"total":96636764160,"available":95991697408,"internally_reserved":0,"allocated":332058624,"data_stored":331894585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,4,2,0,3,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.20100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.76400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":14.079000000000001},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":10.861000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562008,"kb_used_data":255824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93809832,"statfs":{"total":96636764160,"available":96061267968,"internally_reserved":0,"allocated":261963776,"data_stored":261796591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.35499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":3.8450000000000002},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":3.4119999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":3.8079999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626736,"kb_used_data":324776,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93745104,"statfs":{"total":96636764160,"available":95994986496,"internally_reserved":0,"allocated":332570624,"data_stored":332403439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,3,4,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.255},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.35799999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.307}]},{"osd":3,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.645,"5min":1.661,"15min":0.99399999999999999},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":7.2039999999999997,"5min":20.948,"15min":20.948},"last":0.27200000000000002},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.3400000000000001,"15min":0.75900000000000001},"min":{"1min":0.28799999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.5229999999999997,"5min":11.6,"15min":11.6},"last":0.34200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331497472,"data_stored":331497472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261591040,"data_stored":261591040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330911744,"data_stored":330911744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331112448,"data_stored":331112448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:39.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:10:39.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6827", 'reported_seq': 6883, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.826923+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:10:37.826923+0000', 'last_peered': '2024-09-15T03:10:37.826923+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:10:37.826923+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6827, 'log_dups_size': 0, 'ondisk_log_size': 6827, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41639936, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6827, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3113}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:39.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:10:39.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6576", 'reported_seq': 6650, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.430258+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:10:37.430258+0000', 'last_peered': '2024-09-15T03:10:37.430258+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:10:37.430258+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6576, 'log_dups_size': 0, 'ondisk_log_size': 6576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39728754, 'num_objects': 4850, 'num_object_clones': 0, 'num_object_copies': 19400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1995, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6574, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2857}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7044", 'reported_seq': 7142, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.942258+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:10:37.942258+0000', 'last_peered': '2024-09-15T03:10:37.942258+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:10:37.942258+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7044, 'log_dups_size': 0, 'ondisk_log_size': 7044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42400593, 'num_objects': 5177, 'num_object_clones': 0, 'num_object_copies': 20708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2096, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5177, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7043, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3082}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6649", 'reported_seq': 8149, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.442558+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:10:37.442558+0000', 'last_peered': '2024-09-15T03:10:37.442558+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:10:37.442558+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6649, 'log_dups_size': 0, 'ondisk_log_size': 6649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40785522, 'num_objects': 4979, 'num_object_clones': 0, 'num_object_copies': 19916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1225, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6647, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 720, 'num_bytes_recovered': 5832704, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6983", 'reported_seq': 7061, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.799970+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:10:37.799970+0000', 'last_peered': '2024-09-15T03:10:37.799970+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:10:37.799970+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6983, 'log_dups_size': 0, 'ondisk_log_size': 6983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41760370, 'num_objects': 5098, 'num_object_clones': 0, 'num_object_copies': 20392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2099, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6981, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3001}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6929", 'reported_seq': 6989, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.736215+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:10:37.736215+0000', 'last_peered': '2024-09-15T03:10:37.736215+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:10:37.736215+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6929, 'log_dups_size': 0, 'ondisk_log_size': 6929, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41525248, 'num_objects': 5069, 'num_object_clones': 0, 'num_object_copies': 20276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6929, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3020}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:10:39.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6888", 'reported_seq': 6967, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.348012+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:10:37.348012+0000', 'last_peered': '2024-09-15T03:10:37.348012+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:10:37.348012+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6888, 'log_dups_size': 0, 'ondisk_log_size': 6888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41530994, 'num_objects': 5070, 'num_object_clones': 0, 'num_object_copies': 20280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2048, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6886, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:39.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:10:39.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6951", 'reported_seq': 7030, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.233649+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:10:37.233649+0000', 'last_peered': '2024-09-15T03:10:37.233649+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:10:37.233649+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6951, 'log_dups_size': 0, 'ondisk_log_size': 6951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42055282, 'num_objects': 5134, 'num_object_clones': 0, 'num_object_copies': 20536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6949, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3060}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:39.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:10:39.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6906", 'reported_seq': 6981, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.841505+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:10:37.841505+0000', 'last_peered': '2024-09-15T03:10:37.841505+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:10:37.841505+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6906, 'log_dups_size': 0, 'ondisk_log_size': 6906, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41479396, 'num_objects': 5064, 'num_object_clones': 0, 'num_object_copies': 20256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6903, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3030}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:39.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:10:39.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6945", 'reported_seq': 7042, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:38.032788+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:10:38.032788+0000', 'last_peered': '2024-09-15T03:10:38.032788+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:10:38.032788+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6945, 'log_dups_size': 0, 'ondisk_log_size': 6945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41145970, 'num_objects': 5023, 'num_object_clones': 0, 'num_object_copies': 20092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5023, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6943, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2971}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:39.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:10:39.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6848", 'reported_seq': 6914, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.762472+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:10:37.762472+0000', 'last_peered': '2024-09-15T03:10:37.762472+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:10:37.762472+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6848, 'log_dups_size': 0, 'ondisk_log_size': 6848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41243051, 'num_objects': 5035, 'num_object_clones': 0, 'num_object_copies': 20140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2077, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6845, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:39.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:10:39.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6963", 'reported_seq': 7052, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.660614+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:10:37.660614+0000', 'last_peered': '2024-09-15T03:10:37.660614+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:10:37.660614+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6963, 'log_dups_size': 0, 'ondisk_log_size': 6963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42351417, 'num_objects': 5170, 'num_object_clones': 0, 'num_object_copies': 20680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2035, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6962, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3136}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:39.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:10:39.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6718", 'reported_seq': 6788, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.746649+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:10:37.746649+0000', 'last_peered': '2024-09-15T03:10:37.746649+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:10:37.746649+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6718, 'log_dups_size': 0, 'ondisk_log_size': 6718, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40662642, 'num_objects': 4964, 'num_object_clones': 0, 'num_object_copies': 19856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2029, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6716, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:39.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:10:39.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6816", 'reported_seq': 6905, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.847808+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:10:37.847808+0000', 'last_peered': '2024-09-15T03:10:37.847808+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:10:37.847808+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6816, 'log_dups_size': 0, 'ondisk_log_size': 6816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41506418, 'num_objects': 5067, 'num_object_clones': 0, 'num_object_copies': 20268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1987, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6814, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3082}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:39.237 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:10:39.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6969", 'reported_seq': 7052, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:37.459056+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:10:37.459056+0000', 'last_peered': '2024-09-15T03:10:37.459056+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:10:37.459056+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6969, 'log_dups_size': 0, 'ondisk_log_size': 6969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41285234, 'num_objects': 5040, 'num_object_clones': 0, 'num_object_copies': 20160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2075, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5040, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6967, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2967}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:39.238 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:10:41.170 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:41.171 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29852/323508 objects degraded (9.228%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":364,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80877,"data_bytes":662501972,"bytes_used":2498932736,"bytes_avail":384048123904,"bytes_total":386547056640,"degraded_objects":29852,"degraded_total":323508,"degraded_ratio":0.092275925170320364,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":125534,"recovering_keys_per_sec":0,"num_objects_recovered":195,"num_bytes_recovered":1507328,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":296},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:10:41.217 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:10:41.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:42.902 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:42.934 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:42.967 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-15T03:10:42.783593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659626580,"num_objects":80526,"num_object_clones":0,"num_object_copies":322104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110181,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2892,"num_bytes_recovered":22659072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110208,"ondisk_log_size":110208,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2444204,"kb_used_data":1224192,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375043156,"statfs":{"total":386547056640,"available":384044191744,"internally_reserved":0,"allocated":1253572608,"data_stored":1252907014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,5,12,11,12,6,5,0,0,2,0,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22093824,"num_objects":-2697,"num_object_clones":0,"num_object_copies":-10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2697,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007180"},"pg_stats":[{"pgid":"1.f","version":"33'6827","reported_seq":6883,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.826923+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:37.826923+0000","last_peered":"2024-09-15T03:10:37.826923+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:37.826923+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6827,"log_dups_size":0,"ondisk_log_size":6827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41639936,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6827,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6651","reported_seq":6724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.455893+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:42.455893+0000","last_peered":"2024-09-15T03:10:42.455893+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:42.455893+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39114354,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7044","reported_seq":7142,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.942258+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:37.942258+0000","last_peered":"2024-09-15T03:10:37.942258+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:37.942258+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7044,"log_dups_size":0,"ondisk_log_size":7044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42400593,"num_objects":5177,"num_object_clones":0,"num_object_copies":20708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7043,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3082}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6716","reported_seq":8342,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.570271+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:42.570271+0000","last_peered":"2024-09-15T03:10:42.570271+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:42.570271+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236658,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6983","reported_seq":7061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.799970+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:37.799970+0000","last_peered":"2024-09-15T03:10:37.799970+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:37.799970+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41760370,"num_objects":5098,"num_object_clones":0,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6981,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3001}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6929","reported_seq":6989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.736215+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:37.736215+0000","last_peered":"2024-09-15T03:10:37.736215+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:37.736215+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6929,"log_dups_size":0,"ondisk_log_size":6929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41525248,"num_objects":5069,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6977","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.546856+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:42.546856+0000","last_peered":"2024-09-15T03:10:42.546856+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:42.546856+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40801906,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2935}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7013","reported_seq":7092,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.563375+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:42.563375+0000","last_peered":"2024-09-15T03:10:42.563375+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:42.563375+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7013,"log_dups_size":0,"ondisk_log_size":7013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41547378,"num_objects":5072,"num_object_clones":0,"num_object_copies":20288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7011,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6906","reported_seq":6981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.841505+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:37.841505+0000","last_peered":"2024-09-15T03:10:37.841505+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:37.841505+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41479396,"num_objects":5064,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6903,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6945","reported_seq":7042,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:38.032788+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:38.032788+0000","last_peered":"2024-09-15T03:10:38.032788+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:38.032788+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41145970,"num_objects":5023,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6848","reported_seq":6914,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.762472+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:37.762472+0000","last_peered":"2024-09-15T03:10:37.762472+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:37.762472+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243051,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6845,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6845","reported_seq":10884,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:38.141308+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:38.141308+0000","last_peered":"2024-09-15T03:10:38.141308+0000","last_clean":"2024-09-15T03:10:38.141308+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:38.141308+0000","last_undegraded":"2024-09-15T03:10:38.141308+0000","last_fullsized":"2024-09-15T03:10:38.141308+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41401145,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6844,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'6963","reported_seq":7052,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.660614+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:37.660614+0000","last_peered":"2024-09-15T03:10:37.660614+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:37.660614+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42351417,"num_objects":5170,"num_object_clones":0,"num_object_copies":20680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3136}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6718","reported_seq":6788,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.746649+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:37.746649+0000","last_peered":"2024-09-15T03:10:37.746649+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:37.746649+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40662642,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6816","reported_seq":6905,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:37.847808+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:37.847808+0000","last_peered":"2024-09-15T03:10:37.847808+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:37.847808+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6816,"log_dups_size":0,"ondisk_log_size":6816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41506418,"num_objects":5067,"num_object_clones":0,"num_object_copies":20268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3082}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7027","reported_seq":7110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.340617+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:42.340617+0000","last_peered":"2024-09-15T03:10:42.340617+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:42.340617+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7027,"log_dups_size":0,"ondisk_log_size":7027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40810098,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7025,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":659626580,"num_objects":80526,"num_object_clones":0,"num_object_copies":322104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110181,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2892,"num_bytes_recovered":22659072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1250390016,"data_stored":1250390016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110208,"ondisk_log_size":110208,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621672,"kb_used_data":323796,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93750168,"statfs":{"total":96636764160,"available":96000172032,"internally_reserved":0,"allocated":331567104,"data_stored":331399919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,5,1,2,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":3.23,"5min":2.0449999999999999,"15min":1.8109999999999999},"min":{"1min":0.19,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":13.266999999999999,"5min":17.408999999999999,"15min":17.408999999999999},"last":1.865},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.2269999999999999,"15min":2.1200000000000001},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.188},"max":{"1min":13.052,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.8460000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:09:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9510000000000001,"5min":1.7889999999999999,"15min":0.95099999999999996},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":24.946000000000002,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":4.1870000000000003,"5min":1.823,"15min":1.2310000000000001},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.188},"max":{"1min":21.398,"5min":21.398,"15min":21.398},"last":0.307}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629948,"kb_used_data":324276,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93741892,"statfs":{"total":96636764160,"available":95991697408,"internally_reserved":0,"allocated":332058624,"data_stored":331894585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,4,2,0,3,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.20100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.76400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":14.079000000000001},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":10.861000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562008,"kb_used_data":255824,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93809832,"statfs":{"total":96636764160,"available":96061267968,"internally_reserved":0,"allocated":261963776,"data_stored":261796591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,3,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.35499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":3.8450000000000002},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":3.4119999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":3.8079999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630576,"kb_used_data":320296,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93741264,"statfs":{"total":96636764160,"available":95991054336,"internally_reserved":0,"allocated":327983104,"data_stored":327815919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,4,1,5,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":6.5279999999999996},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":9.9969999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":4.665},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":4.8979999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.51400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326774784,"data_stored":326774784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261591040,"data_stored":261591040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330911744,"data_stored":330911744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":331112448,"data_stored":331112448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:45.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:47.390 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:47.394 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:47.409 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-15T03:10:46.784655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653122132,"num_objects":79732,"num_object_clones":0,"num_object_copies":318928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110975,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":22659072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111002,"ondisk_log_size":111002,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2451412,"kb_used_data":1214772,"kb_used_omap":14,"kb_used_meta":1236593,"kb_avail":375035948,"statfs":{"total":386547056640,"available":384036810752,"internally_reserved":0,"allocated":1243926528,"data_stored":1243260934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266271259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,9,13,9,2,6,2,0,0,0,2,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18964480,"num_objects":-2315,"num_object_clones":0,"num_object_copies":-9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005534"},"pg_stats":[{"pgid":"1.f","version":"33'6893","reported_seq":6949,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.603901+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:42.603901+0000","last_peered":"2024-09-15T03:10:42.603901+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:42.603901+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6893,"log_dups_size":0,"ondisk_log_size":6893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41099264,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6893,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6651","reported_seq":6724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.455893+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:42.455893+0000","last_peered":"2024-09-15T03:10:42.455893+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:42.455893+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39114354,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7124","reported_seq":7224,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:43.057867+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:43.057867+0000","last_peered":"2024-09-15T03:10:43.057867+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:43.057867+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41745233,"num_objects":5097,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7123,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6716","reported_seq":8342,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.570271+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:42.570271+0000","last_peered":"2024-09-15T03:10:42.570271+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:42.570271+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236658,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7055","reported_seq":7133,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.829153+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:42.829153+0000","last_peered":"2024-09-15T03:10:42.829153+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:42.829153+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41170546,"num_objects":5026,"num_object_clones":0,"num_object_copies":20104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7004","reported_seq":7064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.914129+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:42.914129+0000","last_peered":"2024-09-15T03:10:42.914129+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:42.914129+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7004,"log_dups_size":0,"ondisk_log_size":7004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40910848,"num_objects":4994,"num_object_clones":0,"num_object_copies":19976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7004,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2945}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6977","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.546856+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:42.546856+0000","last_peered":"2024-09-15T03:10:42.546856+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:42.546856+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40801906,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2935}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7013","reported_seq":7092,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.563375+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:42.563375+0000","last_peered":"2024-09-15T03:10:42.563375+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:42.563375+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7013,"log_dups_size":0,"ondisk_log_size":7013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41547378,"num_objects":5072,"num_object_clones":0,"num_object_copies":20288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7011,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6987","reported_seq":7062,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.732017+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:42.732017+0000","last_peered":"2024-09-15T03:10:42.732017+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:42.732017+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40815844,"num_objects":4983,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7018","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:43.021888+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:43.021888+0000","last_peered":"2024-09-15T03:10:43.021888+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:43.021888+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7018,"log_dups_size":0,"ondisk_log_size":7018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40547954,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6923","reported_seq":6991,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.884969+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:42.884969+0000","last_peered":"2024-09-15T03:10:42.884969+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:42.884969+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40628651,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6920,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6923","reported_seq":10962,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:43.210568+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:43.210568+0000","last_peered":"2024-09-15T03:10:43.210568+0000","last_clean":"2024-09-15T03:10:43.210568+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:43.210568+0000","last_undegraded":"2024-09-15T03:10:43.210568+0000","last_fullsized":"2024-09-15T03:10:43.210568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40762169,"num_objects":4976,"num_object_clones":0,"num_object_copies":19904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6922,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7042","reported_seq":7131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.956995+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:42.956995+0000","last_peered":"2024-09-15T03:10:42.956995+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:42.956995+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7042,"log_dups_size":0,"ondisk_log_size":7042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41704249,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3057}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6777","reported_seq":6847,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.677005+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:42.677005+0000","last_peered":"2024-09-15T03:10:42.677005+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:42.677005+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40179314,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2878}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6872","reported_seq":6963,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.912472+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:42.912472+0000","last_peered":"2024-09-15T03:10:42.912472+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:42.912472+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047666,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6870,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7027","reported_seq":7110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.340617+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:42.340617+0000","last_peered":"2024-09-15T03:10:42.340617+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:42.340617+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7027,"log_dups_size":0,"ondisk_log_size":7027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40810098,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7025,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":653122132,"num_objects":79732,"num_object_clones":0,"num_object_copies":318928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110975,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":22659072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1240842240,"data_stored":1240842240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111002,"ondisk_log_size":111002,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625388,"kb_used_data":319204,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93746452,"statfs":{"total":96636764160,"available":95996366848,"internally_reserved":0,"allocated":326864896,"data_stored":326697711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,3,3,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.185}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.248},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":7.7709999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629636,"kb_used_data":319804,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93742204,"statfs":{"total":96636764160,"available":95992016896,"internally_reserved":0,"allocated":327479296,"data_stored":327315257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,0,1,1,2,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":31.175000000000001},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.316}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.39300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":6.5410000000000004},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":6.5270000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565812,"kb_used_data":255468,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93806028,"statfs":{"total":96636764160,"available":96057372672,"internally_reserved":0,"allocated":261599232,"data_stored":261432047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":4.3849999999999998},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.24099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":1.79},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":1.8779999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":1.994}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630576,"kb_used_data":320296,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93741264,"statfs":{"total":96636764160,"available":95991054336,"internally_reserved":0,"allocated":327983104,"data_stored":327815919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,4,1,5,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":6.5279999999999996},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":9.9969999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":4.665},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":4.8979999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.51400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326774784,"data_stored":326774784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261259264,"data_stored":261259264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326311936,"data_stored":326311936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":326496256,"data_stored":326496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:47.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:48.470 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:48.496 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:48.534 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-15T03:10:46.784655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653122132,"num_objects":79732,"num_object_clones":0,"num_object_copies":318928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110975,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":22659072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111002,"ondisk_log_size":111002,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2451412,"kb_used_data":1214772,"kb_used_omap":14,"kb_used_meta":1236593,"kb_avail":375035948,"statfs":{"total":386547056640,"available":384036810752,"internally_reserved":0,"allocated":1243926528,"data_stored":1243260934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266271259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,9,13,9,2,6,2,0,0,0,2,2,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18964480,"num_objects":-2315,"num_object_clones":0,"num_object_copies":-9260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005534"},"pg_stats":[{"pgid":"1.f","version":"33'6893","reported_seq":6949,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.603901+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:42.603901+0000","last_peered":"2024-09-15T03:10:42.603901+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:42.603901+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6893,"log_dups_size":0,"ondisk_log_size":6893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41099264,"num_objects":5017,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6893,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6651","reported_seq":6724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.455893+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:42.455893+0000","last_peered":"2024-09-15T03:10:42.455893+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:42.455893+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6651,"log_dups_size":0,"ondisk_log_size":6651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39114354,"num_objects":4775,"num_object_clones":0,"num_object_copies":19100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2782}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7124","reported_seq":7224,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:43.057867+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:43.057867+0000","last_peered":"2024-09-15T03:10:43.057867+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:43.057867+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7124,"log_dups_size":0,"ondisk_log_size":7124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41745233,"num_objects":5097,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7123,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3003}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6716","reported_seq":8342,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.570271+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:42.570271+0000","last_peered":"2024-09-15T03:10:42.570271+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:42.570271+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236658,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6714,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":782,"num_bytes_recovered":6340608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7055","reported_seq":7133,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.829153+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:42.829153+0000","last_peered":"2024-09-15T03:10:42.829153+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:42.829153+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41170546,"num_objects":5026,"num_object_clones":0,"num_object_copies":20104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7004","reported_seq":7064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.914129+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:42.914129+0000","last_peered":"2024-09-15T03:10:42.914129+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:42.914129+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7004,"log_dups_size":0,"ondisk_log_size":7004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40910848,"num_objects":4994,"num_object_clones":0,"num_object_copies":19976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2049,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7004,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2945}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6977","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.546856+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:42.546856+0000","last_peered":"2024-09-15T03:10:42.546856+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:42.546856+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6977,"log_dups_size":0,"ondisk_log_size":6977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40801906,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2935}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7013","reported_seq":7092,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.563375+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:42.563375+0000","last_peered":"2024-09-15T03:10:42.563375+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:42.563375+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7013,"log_dups_size":0,"ondisk_log_size":7013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41547378,"num_objects":5072,"num_object_clones":0,"num_object_copies":20288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7011,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6987","reported_seq":7062,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.732017+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:42.732017+0000","last_peered":"2024-09-15T03:10:42.732017+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:42.732017+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40815844,"num_objects":4983,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7018","reported_seq":7115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:43.021888+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:43.021888+0000","last_peered":"2024-09-15T03:10:43.021888+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:43.021888+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7018,"log_dups_size":0,"ondisk_log_size":7018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40547954,"num_objects":4950,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2898}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6923","reported_seq":6991,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.884969+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:42.884969+0000","last_peered":"2024-09-15T03:10:42.884969+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:42.884969+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40628651,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2077,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6920,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6923","reported_seq":10962,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:43.210568+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:43.210568+0000","last_peered":"2024-09-15T03:10:43.210568+0000","last_clean":"2024-09-15T03:10:43.210568+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:43.210568+0000","last_undegraded":"2024-09-15T03:10:43.210568+0000","last_fullsized":"2024-09-15T03:10:43.210568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40762169,"num_objects":4976,"num_object_clones":0,"num_object_copies":19904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6922,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7042","reported_seq":7131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.956995+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:42.956995+0000","last_peered":"2024-09-15T03:10:42.956995+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:42.956995+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7042,"log_dups_size":0,"ondisk_log_size":7042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41704249,"num_objects":5091,"num_object_clones":0,"num_object_copies":20364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3057}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6777","reported_seq":6847,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.677005+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:42.677005+0000","last_peered":"2024-09-15T03:10:42.677005+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:42.677005+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40179314,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2029,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2878}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6872","reported_seq":6963,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.912472+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:42.912472+0000","last_peered":"2024-09-15T03:10:42.912472+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:42.912472+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41047666,"num_objects":5011,"num_object_clones":0,"num_object_copies":20044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6870,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3027}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7027","reported_seq":7110,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:42.340617+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:42.340617+0000","last_peered":"2024-09-15T03:10:42.340617+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:42.340617+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7027,"log_dups_size":0,"ondisk_log_size":7027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40810098,"num_objects":4982,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7025,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2909}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":653122132,"num_objects":79732,"num_object_clones":0,"num_object_copies":318928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110975,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2894,"num_bytes_recovered":22659072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1240842240,"data_stored":1240842240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111002,"ondisk_log_size":111002,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625388,"kb_used_data":319204,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93746452,"statfs":{"total":96636764160,"available":95996366848,"internally_reserved":0,"allocated":326864896,"data_stored":326697711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,3,3,0,0,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.185}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.248},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":7.7709999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629636,"kb_used_data":319804,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93742204,"statfs":{"total":96636764160,"available":95992016896,"internally_reserved":0,"allocated":327479296,"data_stored":327315257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,0,1,1,2,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":31.175000000000001},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.316}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.39300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":6.5410000000000004},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":6.5270000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565812,"kb_used_data":255468,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93806028,"statfs":{"total":96636764160,"available":96057372672,"internally_reserved":0,"allocated":261599232,"data_stored":261432047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":4.3849999999999998},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.24099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":1.79},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":1.8779999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":1.994}]}]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630576,"kb_used_data":320296,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93741264,"statfs":{"total":96636764160,"available":95991054336,"internally_reserved":0,"allocated":327983104,"data_stored":327815919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,4,1,5,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":6.5279999999999996},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":9.9969999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":4.665},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":4.8979999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.51400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326774784,"data_stored":326774784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261259264,"data_stored":261259264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326311936,"data_stored":326311936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":326496256,"data_stored":326496256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:48.535 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:10:48.535 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6893", 'reported_seq': 6949, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.603901+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:10:42.603901+0000', 'last_peered': '2024-09-15T03:10:42.603901+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:10:42.603901+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6893, 'log_dups_size': 0, 'ondisk_log_size': 6893, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41099264, 'num_objects': 5017, 'num_object_clones': 0, 'num_object_copies': 20068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1970, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6893, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6651", 'reported_seq': 6724, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.455893+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:10:42.455893+0000', 'last_peered': '2024-09-15T03:10:42.455893+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:10:42.455893+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6651, 'log_dups_size': 0, 'ondisk_log_size': 6651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39114354, 'num_objects': 4775, 'num_object_clones': 0, 'num_object_copies': 19100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1995, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6649, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2782}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7124", 'reported_seq': 7224, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:43.057867+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:10:43.057867+0000', 'last_peered': '2024-09-15T03:10:43.057867+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:10:43.057867+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7124, 'log_dups_size': 0, 'ondisk_log_size': 7124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41745233, 'num_objects': 5097, 'num_object_clones': 0, 'num_object_copies': 20388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5097, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7123, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3003}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6716", 'reported_seq': 8342, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.570271+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:10:42.570271+0000', 'last_peered': '2024-09-15T03:10:42.570271+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:10:42.570271+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6716, 'log_dups_size': 0, 'ondisk_log_size': 6716, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40236658, 'num_objects': 4912, 'num_object_clones': 0, 'num_object_copies': 19648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1163, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6714, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 782, 'num_bytes_recovered': 6340608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7055", 'reported_seq': 7133, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.829153+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:10:42.829153+0000', 'last_peered': '2024-09-15T03:10:42.829153+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:10:42.829153+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7055, 'log_dups_size': 0, 'ondisk_log_size': 7055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41170546, 'num_objects': 5026, 'num_object_clones': 0, 'num_object_copies': 20104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2099, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7053, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2929}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:10:48.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7004", 'reported_seq': 7064, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.914129+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:10:42.914129+0000', 'last_peered': '2024-09-15T03:10:42.914129+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:10:42.914129+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7004, 'log_dups_size': 0, 'ondisk_log_size': 7004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40910848, 'num_objects': 4994, 'num_object_clones': 0, 'num_object_copies': 19976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2049, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7004, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2945}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6977", 'reported_seq': 7056, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.546856+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:10:42.546856+0000', 'last_peered': '2024-09-15T03:10:42.546856+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:10:42.546856+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6977, 'log_dups_size': 0, 'ondisk_log_size': 6977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40801906, 'num_objects': 4981, 'num_object_clones': 0, 'num_object_copies': 19924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2048, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6975, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2935}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7013", 'reported_seq': 7092, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.563375+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:10:42.563375+0000', 'last_peered': '2024-09-15T03:10:42.563375+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:10:42.563375+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7013, 'log_dups_size': 0, 'ondisk_log_size': 7013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41547378, 'num_objects': 5072, 'num_object_clones': 0, 'num_object_copies': 20288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5072, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7011, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6987", 'reported_seq': 7062, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.732017+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:10:42.732017+0000', 'last_peered': '2024-09-15T03:10:42.732017+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:10:42.732017+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6987, 'log_dups_size': 0, 'ondisk_log_size': 6987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40815844, 'num_objects': 4983, 'num_object_clones': 0, 'num_object_copies': 19932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2037, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6984, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2949}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7018", 'reported_seq': 7115, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:43.021888+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:10:43.021888+0000', 'last_peered': '2024-09-15T03:10:43.021888+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:10:43.021888+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7018, 'log_dups_size': 0, 'ondisk_log_size': 7018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40547954, 'num_objects': 4950, 'num_object_clones': 0, 'num_object_copies': 19800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7016, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2898}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:10:48.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'6923", 'reported_seq': 6991, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.884969+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:10:42.884969+0000', 'last_peered': '2024-09-15T03:10:42.884969+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:10:42.884969+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6923, 'log_dups_size': 0, 'ondisk_log_size': 6923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40628651, 'num_objects': 4960, 'num_object_clones': 0, 'num_object_copies': 19840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2077, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6920, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2886}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:48.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:10:48.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7042", 'reported_seq': 7131, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.956995+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:10:42.956995+0000', 'last_peered': '2024-09-15T03:10:42.956995+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:10:42.956995+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7042, 'log_dups_size': 0, 'ondisk_log_size': 7042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41704249, 'num_objects': 5091, 'num_object_clones': 0, 'num_object_copies': 20364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2035, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7041, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:48.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:10:48.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6777", 'reported_seq': 6847, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.677005+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:10:42.677005+0000', 'last_peered': '2024-09-15T03:10:42.677005+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:10:42.677005+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6777, 'log_dups_size': 0, 'ondisk_log_size': 6777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40179314, 'num_objects': 4905, 'num_object_clones': 0, 'num_object_copies': 19620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2029, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6775, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2878}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:48.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:10:48.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6872", 'reported_seq': 6963, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.912472+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:10:42.912472+0000', 'last_peered': '2024-09-15T03:10:42.912472+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:10:42.912472+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6872, 'log_dups_size': 0, 'ondisk_log_size': 6872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41047666, 'num_objects': 5011, 'num_object_clones': 0, 'num_object_copies': 20044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6870, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3027}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:48.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:10:48.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7027", 'reported_seq': 7110, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:42.340617+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:10:42.340617+0000', 'last_peered': '2024-09-15T03:10:42.340617+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:10:42.340617+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7027, 'log_dups_size': 0, 'ondisk_log_size': 7027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40810098, 'num_objects': 4982, 'num_object_clones': 0, 'num_object_copies': 19928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2075, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7025, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2909}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:48.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:10:50.033 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:50.033 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29721/314272 objects degraded (9.457%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":373,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78568,"data_bytes":643586644,"bytes_used":2468249600,"bytes_avail":384078807040,"bytes_total":386547056640,"degraded_objects":29721,"degraded_total":314272,"degraded_ratio":0.094570944913959884,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":118729,"recovering_keys_per_sec":0,"num_objects_recovered":194,"num_bytes_recovered":1425408,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":289},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:10:50.052 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:10:50.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:51.308 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:51.312 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:51.332 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-15T03:10:50.786777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":643586644,"num_objects":78568,"num_object_clones":0,"num_object_copies":314272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78568,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112139,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2963,"num_bytes_recovered":23134208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112166,"ondisk_log_size":112166,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2410400,"kb_used_data":1198700,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375076960,"statfs":{"total":386547056640,"available":384078807040,"internally_reserved":0,"allocated":1227468800,"data_stored":1226803206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,6,27,5,10,9,3,0,0,1,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18915328,"num_objects":-2309,"num_object_clones":0,"num_object_copies":-9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2309,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006575"},"pg_stats":[{"pgid":"1.f","version":"33'6973","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.928896+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:47.928896+0000","last_peered":"2024-09-15T03:10:47.928896+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:47.928896+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6973,"log_dups_size":0,"ondisk_log_size":6973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40443904,"num_objects":4937,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6973,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6733","reported_seq":6807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.441199+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:47.441199+0000","last_peered":"2024-09-15T03:10:47.441199+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:47.441199+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6733,"log_dups_size":0,"ondisk_log_size":6733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38442610,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6731,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2700}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7190","reported_seq":7290,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.805031+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:47.805031+0000","last_peered":"2024-09-15T03:10:47.805031+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:47.805031+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7190,"log_dups_size":0,"ondisk_log_size":7190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41204561,"num_objects":5031,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7189,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6776","reported_seq":8522,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.477174+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:47.477174+0000","last_peered":"2024-09-15T03:10:47.477174+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:47.477174+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6776,"log_dups_size":0,"ondisk_log_size":6776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39745138,"num_objects":4852,"num_object_clones":0,"num_object_copies":19408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6774,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":842,"num_bytes_recovered":6815744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7121","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.929610+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:47.929610+0000","last_peered":"2024-09-15T03:10:47.929610+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:47.929610+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40629874,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7119,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7085","reported_seq":7145,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.846673+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:47.846673+0000","last_peered":"2024-09-15T03:10:47.846673+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:47.846673+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7085,"log_dups_size":0,"ondisk_log_size":7085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40247296,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7085,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2865}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7053","reported_seq":7132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.595275+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:47.595275+0000","last_peered":"2024-09-15T03:10:47.595275+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:47.595275+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7053,"log_dups_size":0,"ondisk_log_size":7053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40179314,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7051,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2859}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7092","reported_seq":7171,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.605128+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:47.605128+0000","last_peered":"2024-09-15T03:10:47.605128+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:47.605128+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7092,"log_dups_size":0,"ondisk_log_size":7092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40900210,"num_objects":4993,"num_object_clones":0,"num_object_copies":19972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7058","reported_seq":7136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.837325+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:47.837325+0000","last_peered":"2024-09-15T03:10:47.837325+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:47.837325+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40234212,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7075","reported_seq":7172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:48.013405+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:48.013405+0000","last_peered":"2024-09-15T03:10:48.013405+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:48.013405+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7075,"log_dups_size":0,"ondisk_log_size":7075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40081010,"num_objects":4893,"num_object_clones":0,"num_object_copies":19572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7073,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2841}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7000","reported_seq":7070,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.928835+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:47.928835+0000","last_peered":"2024-09-15T03:10:47.928835+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:47.928835+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7000,"log_dups_size":0,"ondisk_log_size":7000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39997867,"num_objects":4883,"num_object_clones":0,"num_object_copies":19532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6997,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6998","reported_seq":11037,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:48.145721+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:48.145721+0000","last_peered":"2024-09-15T03:10:48.145721+0000","last_clean":"2024-09-15T03:10:48.145721+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:48.145721+0000","last_undegraded":"2024-09-15T03:10:48.145721+0000","last_fullsized":"2024-09-15T03:10:48.145721+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6998,"log_dups_size":0,"ondisk_log_size":6998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40147769,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6997,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7125","reported_seq":7214,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:48.038787+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:48.038787+0000","last_peered":"2024-09-15T03:10:48.038787+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:48.038787+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7125,"log_dups_size":0,"ondisk_log_size":7125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41024313,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7124,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2974}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6863","reported_seq":6935,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.666468+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:47.666468+0000","last_peered":"2024-09-15T03:10:47.666468+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:47.666468+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6863,"log_dups_size":0,"ondisk_log_size":6863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474802,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6861,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2793}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6941","reported_seq":7032,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:48.044748+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:48.044748+0000","last_peered":"2024-09-15T03:10:48.044748+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:48.044748+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40482418,"num_objects":4942,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6939,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2958}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7083","reported_seq":7168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:47.381712+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:47.381712+0000","last_peered":"2024-09-15T03:10:47.381712+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:47.381712+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40351346,"num_objects":4926,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7081,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2854}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":643586644,"num_objects":78568,"num_object_clones":0,"num_object_copies":314272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78568,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112139,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2963,"num_bytes_recovered":23134208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1226166272,"data_stored":1226166272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112166,"ondisk_log_size":112166,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624880,"kb_used_data":314536,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93746960,"statfs":{"total":96636764160,"available":95996887040,"internally_reserved":0,"allocated":322084864,"data_stored":321917679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,16,1,0,4,0,0,0,0,0,0,2],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.193},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":9.4030000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":3.0939999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":3.1669999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":3.0459999999999998},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":4.1509999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611712,"kb_used_data":314348,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93760128,"statfs":{"total":96636764160,"available":96010371072,"internally_reserved":0,"allocated":321892352,"data_stored":321728313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.19900000000000001},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.75700000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.84699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.57299999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557128,"kb_used_data":255104,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93814712,"statfs":{"total":96636764160,"available":96066265088,"internally_reserved":0,"allocated":261226496,"data_stored":261059311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,2,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":3.3159999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":0.372}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.313},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.22700000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616680,"kb_used_data":314712,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93755160,"statfs":{"total":96636764160,"available":96005283840,"internally_reserved":0,"allocated":322265088,"data_stored":322097903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,7,2,4,3,3,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.58799999999999997},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":1.909}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321966080,"data_stored":321966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260947968,"data_stored":260947968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321560576,"data_stored":321560576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":321691648,"data_stored":321691648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:54.334 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:55.620 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:55.666 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-15T03:10:54.788401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634600020,"num_objects":77471,"num_object_clones":0,"num_object_copies":309884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113236,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3047,"num_bytes_recovered":23748608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113263,"ondisk_log_size":113263,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2401372,"kb_used_data":1185532,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375085988,"statfs":{"total":386547056640,"available":384088051712,"internally_reserved":0,"allocated":1213984768,"data_stored":1213319174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,3,14,12,6,1,0,0,0,0,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25026560,"num_objects":-3055,"num_object_clones":0,"num_object_copies":-12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3055,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004807"},"pg_stats":[{"pgid":"1.f","version":"33'7048","reported_seq":7106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.749490+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:52.749490+0000","last_peered":"2024-09-15T03:10:52.749490+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:52.749490+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39829504,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7048,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6806","reported_seq":6880,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.481084+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:52.481084+0000","last_peered":"2024-09-15T03:10:52.481084+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:52.481084+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6806,"log_dups_size":0,"ondisk_log_size":6806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37844594,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6804,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2627}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7251","reported_seq":7351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.945902+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:52.945902+0000","last_peered":"2024-09-15T03:10:52.945902+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:52.945902+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7251,"log_dups_size":0,"ondisk_log_size":7251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40704849,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7250,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6844","reported_seq":8740,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.699484+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:52.699484+0000","last_peered":"2024-09-15T03:10:52.699484+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:52.699484+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39188082,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6842,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":7430144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7181","reported_seq":7267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.879358+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:52.879358+0000","last_peered":"2024-09-15T03:10:52.879358+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:52.879358+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7181,"log_dups_size":0,"ondisk_log_size":7181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40138354,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7179,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2807}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7153","reported_seq":7219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.780153+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:52.780153+0000","last_peered":"2024-09-15T03:10:52.780153+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:52.780153+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39690240,"num_objects":4845,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7153,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7114","reported_seq":7193,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.607078+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:52.607078+0000","last_peered":"2024-09-15T03:10:52.607078+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:52.607078+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7114,"log_dups_size":0,"ondisk_log_size":7114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39679602,"num_objects":4844,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7112,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2798}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7160","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.526677+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:52.526677+0000","last_peered":"2024-09-15T03:10:52.526677+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:52.526677+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7160,"log_dups_size":0,"ondisk_log_size":7160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40343154,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7158,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7145","reported_seq":7221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.811286+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:52.811286+0000","last_peered":"2024-09-15T03:10:52.811286+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:52.811286+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7145,"log_dups_size":0,"ondisk_log_size":7145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39521508,"num_objects":4825,"num_object_clones":0,"num_object_copies":19300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7142","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.972994+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:52.972994+0000","last_peered":"2024-09-15T03:10:52.972994+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:52.972994+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7142,"log_dups_size":0,"ondisk_log_size":7142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39532146,"num_objects":4826,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7140,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2774}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7069","reported_seq":7147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.857102+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:52.857102+0000","last_peered":"2024-09-15T03:10:52.857102+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:52.857102+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7069,"log_dups_size":0,"ondisk_log_size":7069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39432619,"num_objects":4814,"num_object_clones":0,"num_object_copies":19256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7066,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7065","reported_seq":11104,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:53.285797+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:53.285797+0000","last_peered":"2024-09-15T03:10:53.285797+0000","last_clean":"2024-09-15T03:10:53.285797+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:53.285797+0000","last_undegraded":"2024-09-15T03:10:53.285797+0000","last_fullsized":"2024-09-15T03:10:53.285797+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7065,"log_dups_size":0,"ondisk_log_size":7065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39598905,"num_objects":4834,"num_object_clones":0,"num_object_copies":19336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7064,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7190","reported_seq":7279,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:53.141038+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:53.141038+0000","last_peered":"2024-09-15T03:10:53.141038+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:53.141038+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7190,"log_dups_size":0,"ondisk_log_size":7190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40491833,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6932","reported_seq":7003,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.800945+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:52.800945+0000","last_peered":"2024-09-15T03:10:52.800945+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:52.800945+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38909554,"num_objects":4750,"num_object_clones":0,"num_object_copies":19000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6930,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2724}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7015","reported_seq":7105,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:53.040079+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:53.040079+0000","last_peered":"2024-09-15T03:10:53.040079+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:53.040079+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39876210,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2884}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7148","reported_seq":7233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.643497+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:52.643497+0000","last_peered":"2024-09-15T03:10:52.643497+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:52.643497+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39818866,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":634600020,"num_objects":77471,"num_object_clones":0,"num_object_copies":309884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113236,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3047,"num_bytes_recovered":23748608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212538880,"data_stored":1212538880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113263,"ondisk_log_size":113263,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608104,"kb_used_data":310240,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93763736,"statfs":{"total":96636764160,"available":96014065664,"internally_reserved":0,"allocated":317685760,"data_stored":317518575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2,5,3,1,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.25900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":10.926},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.443}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":10.882999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":0.39000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615652,"kb_used_data":309980,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93756188,"statfs":{"total":96636764160,"available":96006336512,"internally_reserved":0,"allocated":317419520,"data_stored":317255481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.41299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.433},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.46000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.35599999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561016,"kb_used_data":254832,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93810824,"statfs":{"total":96636764160,"available":96062283776,"internally_reserved":0,"allocated":260947968,"data_stored":260780783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":2.4300000000000002},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":3.8959999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":26.911000000000001},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":0.82599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.9489999999999998},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":20.681000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616600,"kb_used_data":310480,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93755240,"statfs":{"total":96636764160,"available":96005365760,"internally_reserved":0,"allocated":317931520,"data_stored":317764335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":7.3639999999999999},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.9020000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.58099999999999996},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":6.8200000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":6.9189999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":6.8929999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317501440,"data_stored":317501440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260669440,"data_stored":260669440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317091840,"data_stored":317091840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":317276160,"data_stored":317276160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:55.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:56.994 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:57.013 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:57.023 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-15T03:10:56.790311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634600020,"num_objects":77471,"num_object_clones":0,"num_object_copies":309884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113236,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3047,"num_bytes_recovered":23748608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113263,"ondisk_log_size":113263,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2401372,"kb_used_data":1185532,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375085988,"statfs":{"total":386547056640,"available":384088051712,"internally_reserved":0,"allocated":1213984768,"data_stored":1213319174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,3,14,12,6,1,0,0,0,0,7],"upper_bound":16384},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18522112,"num_objects":-2261,"num_object_clones":0,"num_object_copies":-9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006177"},"pg_stats":[{"pgid":"1.f","version":"33'7048","reported_seq":7106,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.749490+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:52.749490+0000","last_peered":"2024-09-15T03:10:52.749490+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:52.749490+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39829504,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7048,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2893}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6806","reported_seq":6880,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.481084+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:52.481084+0000","last_peered":"2024-09-15T03:10:52.481084+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:52.481084+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6806,"log_dups_size":0,"ondisk_log_size":6806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37844594,"num_objects":4620,"num_object_clones":0,"num_object_copies":18480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6804,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2627}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7251","reported_seq":7351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.945902+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:52.945902+0000","last_peered":"2024-09-15T03:10:52.945902+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:52.945902+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7251,"log_dups_size":0,"ondisk_log_size":7251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40704849,"num_objects":4970,"num_object_clones":0,"num_object_copies":19880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7250,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6844","reported_seq":8740,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.699484+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:52.699484+0000","last_peered":"2024-09-15T03:10:52.699484+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:52.699484+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39188082,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6842,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":917,"num_bytes_recovered":7430144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7181","reported_seq":7267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.879358+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:52.879358+0000","last_peered":"2024-09-15T03:10:52.879358+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:52.879358+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7181,"log_dups_size":0,"ondisk_log_size":7181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40138354,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7179,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2807}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7153","reported_seq":7219,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.780153+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:52.780153+0000","last_peered":"2024-09-15T03:10:52.780153+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:52.780153+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7153,"log_dups_size":0,"ondisk_log_size":7153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39690240,"num_objects":4845,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7153,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7114","reported_seq":7193,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.607078+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:52.607078+0000","last_peered":"2024-09-15T03:10:52.607078+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:52.607078+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7114,"log_dups_size":0,"ondisk_log_size":7114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39679602,"num_objects":4844,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2048,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7112,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2798}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7160","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.526677+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:52.526677+0000","last_peered":"2024-09-15T03:10:52.526677+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:52.526677+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7160,"log_dups_size":0,"ondisk_log_size":7160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40343154,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7158,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2851}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7145","reported_seq":7221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.811286+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:52.811286+0000","last_peered":"2024-09-15T03:10:52.811286+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:52.811286+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7145,"log_dups_size":0,"ondisk_log_size":7145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39521508,"num_objects":4825,"num_object_clones":0,"num_object_copies":19300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7142","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.972994+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:52.972994+0000","last_peered":"2024-09-15T03:10:52.972994+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:52.972994+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7142,"log_dups_size":0,"ondisk_log_size":7142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39532146,"num_objects":4826,"num_object_clones":0,"num_object_copies":19304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2054,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7140,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2774}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7069","reported_seq":7147,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.857102+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:52.857102+0000","last_peered":"2024-09-15T03:10:52.857102+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:52.857102+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7069,"log_dups_size":0,"ondisk_log_size":7069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39432619,"num_objects":4814,"num_object_clones":0,"num_object_copies":19256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7066,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7065","reported_seq":11104,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:53.285797+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:53.285797+0000","last_peered":"2024-09-15T03:10:53.285797+0000","last_clean":"2024-09-15T03:10:53.285797+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:53.285797+0000","last_undegraded":"2024-09-15T03:10:53.285797+0000","last_fullsized":"2024-09-15T03:10:53.285797+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7065,"log_dups_size":0,"ondisk_log_size":7065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39598905,"num_objects":4834,"num_object_clones":0,"num_object_copies":19336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7064,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7190","reported_seq":7279,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:53.141038+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:53.141038+0000","last_peered":"2024-09-15T03:10:53.141038+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:53.141038+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7190,"log_dups_size":0,"ondisk_log_size":7190,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40491833,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7189,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2909}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6932","reported_seq":7003,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.800945+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:52.800945+0000","last_peered":"2024-09-15T03:10:52.800945+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:52.800945+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38909554,"num_objects":4750,"num_object_clones":0,"num_object_copies":19000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6930,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2724}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7015","reported_seq":7105,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:53.040079+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:53.040079+0000","last_peered":"2024-09-15T03:10:53.040079+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:53.040079+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7015,"log_dups_size":0,"ondisk_log_size":7015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39876210,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2884}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7148","reported_seq":7233,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:52.643497+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:52.643497+0000","last_peered":"2024-09-15T03:10:52.643497+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:52.643497+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7148,"log_dups_size":0,"ondisk_log_size":7148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39818866,"num_objects":4861,"num_object_clones":0,"num_object_copies":19444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":634600020,"num_objects":77471,"num_object_clones":0,"num_object_copies":309884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113236,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3047,"num_bytes_recovered":23748608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212538880,"data_stored":1212538880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113263,"ondisk_log_size":113263,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608104,"kb_used_data":310240,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93763736,"statfs":{"total":96636764160,"available":96014065664,"internally_reserved":0,"allocated":317685760,"data_stored":317518575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2,5,3,1,0,0,0,0,6],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.25900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":10.926},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.443}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":10.882999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":0.39000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615652,"kb_used_data":309980,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93756188,"statfs":{"total":96636764160,"available":96006336512,"internally_reserved":0,"allocated":317419520,"data_stored":317255481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.41299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.433},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.46000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.35599999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561016,"kb_used_data":254832,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93810824,"statfs":{"total":96636764160,"available":96062283776,"internally_reserved":0,"allocated":260947968,"data_stored":260780783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":2.4300000000000002},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":3.8959999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":26.911000000000001},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":0.82599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.9489999999999998},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":20.681000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616600,"kb_used_data":310480,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93755240,"statfs":{"total":96636764160,"available":96005365760,"internally_reserved":0,"allocated":317931520,"data_stored":317764335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,2,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":7.3639999999999999},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.9020000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.58099999999999996},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":6.8200000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":6.9189999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":6.8929999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":317501440,"data_stored":317501440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260669440,"data_stored":260669440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317091840,"data_stored":317091840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":317276160,"data_stored":317276160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7048", 'reported_seq': 7106, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.749490+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:10:52.749490+0000', 'last_peered': '2024-09-15T03:10:52.749490+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:10:52.749490+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7048, 'log_dups_size': 0, 'ondisk_log_size': 7048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39829504, 'num_objects': 4862, 'num_object_clones': 0, 'num_object_copies': 19448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7048, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2893}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6806", 'reported_seq': 6880, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.481084+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:10:52.481084+0000', 'last_peered': '2024-09-15T03:10:52.481084+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:10:52.481084+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6806, 'log_dups_size': 0, 'ondisk_log_size': 6806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37844594, 'num_objects': 4620, 'num_object_clones': 0, 'num_object_copies': 18480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1995, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6804, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2627}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7251", 'reported_seq': 7351, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.945902+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:10:52.945902+0000', 'last_peered': '2024-09-15T03:10:52.945902+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:10:52.945902+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7251, 'log_dups_size': 0, 'ondisk_log_size': 7251, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40704849, 'num_objects': 4970, 'num_object_clones': 0, 'num_object_copies': 19880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4970, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7250, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2876}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6844", 'reported_seq': 8740, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.699484+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:10:52.699484+0000', 'last_peered': '2024-09-15T03:10:52.699484+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:10:52.699484+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6844, 'log_dups_size': 0, 'ondisk_log_size': 6844, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39188082, 'num_objects': 4784, 'num_object_clones': 0, 'num_object_copies': 19136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6842, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 917, 'num_bytes_recovered': 7430144, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:57.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7181", 'reported_seq': 7267, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.879358+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:10:52.879358+0000', 'last_peered': '2024-09-15T03:10:52.879358+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:10:52.879358+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7181, 'log_dups_size': 0, 'ondisk_log_size': 7181, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40138354, 'num_objects': 4900, 'num_object_clones': 0, 'num_object_copies': 19600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7179, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2807}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7153", 'reported_seq': 7219, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.780153+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:10:52.780153+0000', 'last_peered': '2024-09-15T03:10:52.780153+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:10:52.780153+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7153, 'log_dups_size': 0, 'ondisk_log_size': 7153, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39690240, 'num_objects': 4845, 'num_object_clones': 0, 'num_object_copies': 19380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7153, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2799}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7114", 'reported_seq': 7193, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.607078+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:10:52.607078+0000', 'last_peered': '2024-09-15T03:10:52.607078+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:10:52.607078+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7114, 'log_dups_size': 0, 'ondisk_log_size': 7114, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39679602, 'num_objects': 4844, 'num_object_clones': 0, 'num_object_copies': 19376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2048, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7112, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2798}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7160", 'reported_seq': 7239, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.526677+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:10:52.526677+0000', 'last_peered': '2024-09-15T03:10:52.526677+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:10:52.526677+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7160, 'log_dups_size': 0, 'ondisk_log_size': 7160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40343154, 'num_objects': 4925, 'num_object_clones': 0, 'num_object_copies': 19700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7158, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7145", 'reported_seq': 7221, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.811286+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:10:52.811286+0000', 'last_peered': '2024-09-15T03:10:52.811286+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:10:52.811286+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7145, 'log_dups_size': 0, 'ondisk_log_size': 7145, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39521508, 'num_objects': 4825, 'num_object_clones': 0, 'num_object_copies': 19300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7142, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2794}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:57.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7142", 'reported_seq': 7239, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.972994+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:10:52.972994+0000', 'last_peered': '2024-09-15T03:10:52.972994+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:10:52.972994+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7142, 'log_dups_size': 0, 'ondisk_log_size': 7142, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39532146, 'num_objects': 4826, 'num_object_clones': 0, 'num_object_copies': 19304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2054, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7140, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2774}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7069", 'reported_seq': 7147, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.857102+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:10:52.857102+0000', 'last_peered': '2024-09-15T03:10:52.857102+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:10:52.857102+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7069, 'log_dups_size': 0, 'ondisk_log_size': 7069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39432619, 'num_objects': 4814, 'num_object_clones': 0, 'num_object_copies': 19256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4814, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7066, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2745}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7190", 'reported_seq': 7279, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:53.141038+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:10:53.141038+0000', 'last_peered': '2024-09-15T03:10:53.141038+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:10:53.141038+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7190, 'log_dups_size': 0, 'ondisk_log_size': 7190, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40491833, 'num_objects': 4943, 'num_object_clones': 0, 'num_object_copies': 19772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2035, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7189, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2909}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6932", 'reported_seq': 7003, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.800945+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:10:52.800945+0000', 'last_peered': '2024-09-15T03:10:52.800945+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:10:52.800945+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6932, 'log_dups_size': 0, 'ondisk_log_size': 6932, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38909554, 'num_objects': 4750, 'num_object_clones': 0, 'num_object_copies': 19000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6930, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2724}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7015", 'reported_seq': 7105, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:53.040079+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:10:53.040079+0000', 'last_peered': '2024-09-15T03:10:53.040079+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:10:53.040079+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7015, 'log_dups_size': 0, 'ondisk_log_size': 7015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39876210, 'num_objects': 4868, 'num_object_clones': 0, 'num_object_copies': 19472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7013, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2884}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:10:57.027 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:10:57.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7148", 'reported_seq': 7233, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:10:52.643497+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:10:52.643497+0000', 'last_peered': '2024-09-15T03:10:52.643497+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:10:52.643497+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7148, 'log_dups_size': 0, 'ondisk_log_size': 7148, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39818866, 'num_objects': 4861, 'num_object_clones': 0, 'num_object_copies': 19444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2074, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7146, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2789}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:10:57.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:10:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:58.126 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29637/309884 objects degraded (9.564%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":381,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77471,"data_bytes":634600020,"bytes_used":2459004928,"bytes_avail":384088051712,"bytes_total":386547056640,"degraded_objects":29637,"degraded_total":309884,"degraded_ratio":0.09563901330820565,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":90747,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":1089536,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":188},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:10:58.185 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:10:58.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:10:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:10:59.712 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-15T03:10:58.790757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":625318484,"num_objects":76338,"num_object_clones":0,"num_object_copies":305352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114369,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3115,"num_bytes_recovered":24223744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114396,"ondisk_log_size":114396,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2395656,"kb_used_data":1171484,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375091704,"statfs":{"total":386547056640,"available":384093904896,"internally_reserved":0,"allocated":1199599616,"data_stored":1198934022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,7,16,22,11,9,3,2,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27803648,"num_objects":-3394,"num_object_clones":0,"num_object_copies":-13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3394,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":1564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006102"},"pg_stats":[{"pgid":"1.f","version":"33'7117","reported_seq":7175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.866690+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:57.866690+0000","last_peered":"2024-09-15T03:10:57.866690+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:57.866690+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39264256,"num_objects":4793,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6880","reported_seq":6960,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.601108+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:10:57.601108+0000","last_peered":"2024-09-15T03:10:57.601108+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:10:57.601108+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37238386,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6878,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7323","reported_seq":7422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:58.206623+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:58.206623+0000","last_peered":"2024-09-15T03:10:58.206623+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:58.206623+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7323,"log_dups_size":0,"ondisk_log_size":7323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40115025,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7322,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6925","reported_seq":8937,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.707789+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:10:57.707789+0000","last_peered":"2024-09-15T03:10:57.707789+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:10:57.707789+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6925,"log_dups_size":0,"ondisk_log_size":6925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38524530,"num_objects":4703,"num_object_clones":0,"num_object_copies":18812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6923,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":975,"num_bytes_recovered":7905280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7241","reported_seq":7327,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.805232+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:57.805232+0000","last_peered":"2024-09-15T03:10:57.805232+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:57.805232+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39646834,"num_objects":4840,"num_object_clones":0,"num_object_copies":19360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7214","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.900080+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:57.900080+0000","last_peered":"2024-09-15T03:10:57.900080+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:57.900080+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7214,"log_dups_size":0,"ondisk_log_size":7214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7214,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7181","reported_seq":7262,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.582288+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:10:57.582288+0000","last_peered":"2024-09-15T03:10:57.582288+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:10:57.582288+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7181,"log_dups_size":0,"ondisk_log_size":7181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39130738,"num_objects":4777,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7179,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2732}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7223","reported_seq":7304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.629290+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:10:57.629290+0000","last_peered":"2024-09-15T03:10:57.629290+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:10:57.629290+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7223,"log_dups_size":0,"ondisk_log_size":7223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39827058,"num_objects":4862,"num_object_clones":0,"num_object_copies":19448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7221,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7211","reported_seq":7294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.852331+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:57.852331+0000","last_peered":"2024-09-15T03:10:57.852331+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:57.852331+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38980836,"num_objects":4759,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7208,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7194","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:58.045645+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:58.045645+0000","last_peered":"2024-09-15T03:10:58.045645+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:58.045645+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7194,"log_dups_size":0,"ondisk_log_size":7194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39106162,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7192,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2725}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7145","reported_seq":7223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.949709+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:57.949709+0000","last_peered":"2024-09-15T03:10:57.949709+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:57.949709+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7145,"log_dups_size":0,"ondisk_log_size":7145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810027,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7144","reported_seq":11183,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:58.274065+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:58.274065+0000","last_peered":"2024-09-15T03:10:58.274065+0000","last_clean":"2024-09-15T03:10:58.274065+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:58.274065+0000","last_undegraded":"2024-09-15T03:10:58.274065+0000","last_fullsized":"2024-09-15T03:10:58.274065+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7144,"log_dups_size":0,"ondisk_log_size":7144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38951737,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7143,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7282","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:58.204988+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:58.204988+0000","last_peered":"2024-09-15T03:10:58.204988+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:58.204988+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39738169,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7281,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7021","reported_seq":7093,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.813100+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:57.813100+0000","last_peered":"2024-09-15T03:10:57.813100+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:57.813100+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38180466,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7019,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2635}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7089","reported_seq":7179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:58.232102+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:58.232102+0000","last_peered":"2024-09-15T03:10:58.232102+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:58.232102+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39270002,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2810}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7206","reported_seq":7291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.547033+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:10:57.547033+0000","last_peered":"2024-09-15T03:10:57.547033+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:10:57.547033+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7206,"log_dups_size":0,"ondisk_log_size":7206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343730,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2074,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":625318484,"num_objects":76338,"num_object_clones":0,"num_object_copies":305352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114369,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3115,"num_bytes_recovered":24223744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198112768,"data_stored":1198112768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114396,"ondisk_log_size":114396,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611880,"kb_used_data":305696,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93759960,"statfs":{"total":96636764160,"available":96010199040,"internally_reserved":0,"allocated":313032704,"data_stored":312865519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,4,5,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":5.7919999999999998},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.34200000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":1.6879999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":5.7450000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":0.36399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615280,"kb_used_data":305448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93756560,"statfs":{"total":96636764160,"available":96006717440,"internally_reserved":0,"allocated":312778752,"data_stored":312614713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,5,6,1,2,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":5.0460000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":2.77},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":2.8079999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":1.911},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.33200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552304,"kb_used_data":254440,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93819536,"statfs":{"total":96636764160,"available":96071204864,"internally_reserved":0,"allocated":260546560,"data_stored":260379375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.37}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":20.055}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.317}]}]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":616192,"kb_used_data":305900,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93755648,"statfs":{"total":96636764160,"available":96005783552,"internally_reserved":0,"allocated":313241600,"data_stored":313074415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,10,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":8.7929999999999993},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.25600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":19.919},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":18.012}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.29399999999999998},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312848384,"data_stored":312848384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260268032,"data_stored":260268032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312377344,"data_stored":312377344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312619008,"data_stored":312619008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:02.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:04.096 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:04.113 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:04.125 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-15T03:11:02.791773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":622606932,"num_objects":76007,"num_object_clones":0,"num_object_copies":304028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114700,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":24879104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114727,"ondisk_log_size":114727,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2382752,"kb_used_data":1166912,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375104608,"statfs":{"total":386547056640,"available":384107118592,"internally_reserved":0,"allocated":1194917888,"data_stored":1194252294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,6,6,14,13,13,7,1,2,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20979712,"num_objects":-2561,"num_object_clones":0,"num_object_copies":-10244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2561,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":239,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004995"},"pg_stats":[{"pgid":"1.f","version":"33'7117","reported_seq":7175,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.866690+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:10:57.866690+0000","last_peered":"2024-09-15T03:10:57.866690+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:10:57.866690+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39264256,"num_objects":4793,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6949","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.629503+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:02.629503+0000","last_peered":"2024-09-15T03:11:02.629503+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:02.629503+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36673138,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7323","reported_seq":7422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:58.206623+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:10:58.206623+0000","last_peered":"2024-09-15T03:10:58.206623+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:10:58.206623+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7323,"log_dups_size":0,"ondisk_log_size":7323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40115025,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7322,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2804}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6986","reported_seq":9158,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.744809+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:02.744809+0000","last_peered":"2024-09-15T03:11:02.744809+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:02.744809+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38024818,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":8560640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7241","reported_seq":7327,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.805232+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:10:57.805232+0000","last_peered":"2024-09-15T03:10:57.805232+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:10:57.805232+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39646834,"num_objects":4840,"num_object_clones":0,"num_object_copies":19360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7239,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7214","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.900080+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:10:57.900080+0000","last_peered":"2024-09-15T03:10:57.900080+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:10:57.900080+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7214,"log_dups_size":0,"ondisk_log_size":7214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":4784,"num_object_clones":0,"num_object_copies":19136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7214,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7247","reported_seq":7334,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.712564+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:02.712564+0000","last_peered":"2024-09-15T03:11:02.712564+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:02.712564+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7247,"log_dups_size":0,"ondisk_log_size":7247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38590066,"num_objects":4711,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7291","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.719664+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:02.719664+0000","last_peered":"2024-09-15T03:11:02.719664+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:02.719664+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7291,"log_dups_size":0,"ondisk_log_size":7291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39270002,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7289,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2723}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7211","reported_seq":7294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.852331+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:10:57.852331+0000","last_peered":"2024-09-15T03:10:57.852331+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:10:57.852331+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38980836,"num_objects":4759,"num_object_clones":0,"num_object_copies":19036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7208,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2729}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7194","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:58.045645+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:10:58.045645+0000","last_peered":"2024-09-15T03:10:58.045645+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:10:58.045645+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7194,"log_dups_size":0,"ondisk_log_size":7194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39106162,"num_objects":4774,"num_object_clones":0,"num_object_copies":19096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7192,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2725}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7145","reported_seq":7223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.949709+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:10:57.949709+0000","last_peered":"2024-09-15T03:10:57.949709+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:10:57.949709+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7145,"log_dups_size":0,"ondisk_log_size":7145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38810027,"num_objects":4738,"num_object_clones":0,"num_object_copies":18952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2669}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7144","reported_seq":11183,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:10:58.274065+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:10:58.274065+0000","last_peered":"2024-09-15T03:10:58.274065+0000","last_clean":"2024-09-15T03:10:58.274065+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:10:58.274065+0000","last_undegraded":"2024-09-15T03:10:58.274065+0000","last_fullsized":"2024-09-15T03:10:58.274065+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7144,"log_dups_size":0,"ondisk_log_size":7144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38951737,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7143,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7282","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:58.204988+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:10:58.204988+0000","last_peered":"2024-09-15T03:10:58.204988+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:10:58.204988+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7282,"log_dups_size":0,"ondisk_log_size":7282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39738169,"num_objects":4851,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7281,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7021","reported_seq":7093,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:57.813100+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:10:57.813100+0000","last_peered":"2024-09-15T03:10:57.813100+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:10:57.813100+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38180466,"num_objects":4661,"num_object_clones":0,"num_object_copies":18644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7019,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2635}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7089","reported_seq":7179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:10:58.232102+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:10:58.232102+0000","last_peered":"2024-09-15T03:10:58.232102+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:10:58.232102+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7089,"log_dups_size":0,"ondisk_log_size":7089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39270002,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2810}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7273","reported_seq":7360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.646818+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:02.646818+0000","last_peered":"2024-09-15T03:11:02.646818+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:02.646818+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38794866,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":622606932,"num_objects":76007,"num_object_clones":0,"num_object_copies":304028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76007,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114700,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":24879104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1193398272,"data_stored":1193398272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114727,"ondisk_log_size":114727,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611880,"kb_used_data":305696,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93759960,"statfs":{"total":96636764160,"available":96010199040,"internally_reserved":0,"allocated":313032704,"data_stored":312865519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3,4,5,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":5.7919999999999998},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.34200000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":1.6879999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":5.7450000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":0.36399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":615280,"kb_used_data":305448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93756560,"statfs":{"total":96636764160,"available":96006717440,"internally_reserved":0,"allocated":312778752,"data_stored":312614713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,3,5,6,1,2,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.23999999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":5.0460000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":2.77},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":2.8079999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":1.911},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.33200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552304,"kb_used_data":254440,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93819536,"statfs":{"total":96636764160,"available":96071204864,"internally_reserved":0,"allocated":260546560,"data_stored":260379375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.37}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":20.055}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.317}]}]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603288,"kb_used_data":301328,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93768552,"statfs":{"total":96636764160,"available":96018997248,"internally_reserved":0,"allocated":308559872,"data_stored":308392687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,4,1,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.19700000000000001},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.9500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":7.1100000000000003},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.34599999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308133888,"data_stored":308133888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260268032,"data_stored":260268032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312377344,"data_stored":312377344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":312619008,"data_stored":312619008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:04.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:05.369 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:05.383 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:05.407 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-15T03:11:04.792287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615914068,"num_objects":75190,"num_object_clones":0,"num_object_copies":300760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75190,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115517,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":24879104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115544,"ondisk_log_size":115544,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2369352,"kb_used_data":1157672,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375118008,"statfs":{"total":386547056640,"available":384120840192,"internally_reserved":0,"allocated":1185456128,"data_stored":1184790534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,5,12,4,8,7,1,0,0,0,1,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24928256,"num_objects":-3043,"num_object_clones":0,"num_object_copies":-12172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005033"},"pg_stats":[{"pgid":"1.f","version":"33'7181","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.828526+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:02.828526+0000","last_peered":"2024-09-15T03:11:02.828526+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:02.828526+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7181,"log_dups_size":0,"ondisk_log_size":7181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38739968,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6949","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.629503+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:02.629503+0000","last_peered":"2024-09-15T03:11:02.629503+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:02.629503+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36673138,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7413","reported_seq":7513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:03.188521+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:03.188521+0000","last_peered":"2024-09-15T03:11:03.188521+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:03.188521+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39377745,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7412,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6986","reported_seq":9158,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.744809+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:02.744809+0000","last_peered":"2024-09-15T03:11:02.744809+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:02.744809+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38024818,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":8560640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7309","reported_seq":7395,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.880171+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:02.880171+0000","last_peered":"2024-09-15T03:11:02.880171+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:02.880171+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7309,"log_dups_size":0,"ondisk_log_size":7309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39089778,"num_objects":4772,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7285","reported_seq":7351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.889353+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:02.889353+0000","last_peered":"2024-09-15T03:11:02.889353+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:02.889353+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7285,"log_dups_size":0,"ondisk_log_size":7285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38608896,"num_objects":4713,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7247","reported_seq":7334,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.712564+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:02.712564+0000","last_peered":"2024-09-15T03:11:02.712564+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:02.712564+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7247,"log_dups_size":0,"ondisk_log_size":7247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38590066,"num_objects":4711,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7291","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.719664+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:02.719664+0000","last_peered":"2024-09-15T03:11:02.719664+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:02.719664+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7291,"log_dups_size":0,"ondisk_log_size":7291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39270002,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7289,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2723}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7290","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.947320+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:02.947320+0000","last_peered":"2024-09-15T03:11:02.947320+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:02.947320+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38333668,"num_objects":4680,"num_object_clones":0,"num_object_copies":18720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7287,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7270","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:03.143059+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:03.143059+0000","last_peered":"2024-09-15T03:11:03.143059+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:03.143059+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38483570,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7232","reported_seq":7310,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.974078+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:02.974078+0000","last_peered":"2024-09-15T03:11:02.974078+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:02.974078+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38097323,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7221","reported_seq":11260,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:03.270903+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:03.270903+0000","last_peered":"2024-09-15T03:11:03.270903+0000","last_clean":"2024-09-15T03:11:03.270903+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:03.270903+0000","last_undegraded":"2024-09-15T03:11:03.270903+0000","last_fullsized":"2024-09-15T03:11:03.270903+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38320953,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7220,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7340","reported_seq":7431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:03.196844+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:03.196844+0000","last_peered":"2024-09-15T03:11:03.196844+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:03.196844+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7340,"log_dups_size":0,"ondisk_log_size":7340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39263033,"num_objects":4793,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7339,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7091","reported_seq":7163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.763853+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:02.763853+0000","last_peered":"2024-09-15T03:11:02.763853+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:02.763853+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7091,"log_dups_size":0,"ondisk_log_size":7091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37607026,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7089,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7166","reported_seq":7257,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:03.239431+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:03.239431+0000","last_peered":"2024-09-15T03:11:03.239431+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:03.239431+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7166,"log_dups_size":0,"ondisk_log_size":7166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38639218,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2733}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7273","reported_seq":7360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.646818+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:02.646818+0000","last_peered":"2024-09-15T03:11:02.646818+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:02.646818+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38794866,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":615914068,"num_objects":75190,"num_object_clones":0,"num_object_copies":300760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75190,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115517,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":24879104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1183838208,"data_stored":1183838208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115544,"ondisk_log_size":115544,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611472,"kb_used_data":301128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93760368,"statfs":{"total":96636764160,"available":96010616832,"internally_reserved":0,"allocated":308355072,"data_stored":308187887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,6,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.0309999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":2.7789999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":2.899}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598328,"kb_used_data":300976,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93773512,"statfs":{"total":96636764160,"available":96024076288,"internally_reserved":0,"allocated":308199424,"data_stored":308035385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,0,1,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.32200000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.371},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.40699999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.439},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.39500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556264,"kb_used_data":254240,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93815576,"statfs":{"total":96636764160,"available":96067149824,"internally_reserved":0,"allocated":260341760,"data_stored":260174575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.37}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":20.055}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.317}]}]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603288,"kb_used_data":301328,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93768552,"statfs":{"total":96636764160,"available":96018997248,"internally_reserved":0,"allocated":308559872,"data_stored":308392687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,4,1,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.19700000000000001},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.9500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":7.1100000000000003},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.34599999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308133888,"data_stored":308133888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260055040,"data_stored":260055040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307716096,"data_stored":307716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307933184,"data_stored":307933184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:05.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:11:05.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7181", 'reported_seq': 7239, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.828526+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:11:02.828526+0000', 'last_peered': '2024-09-15T03:11:02.828526+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:11:02.828526+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7181, 'log_dups_size': 0, 'ondisk_log_size': 7181, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38739968, 'num_objects': 4729, 'num_object_clones': 0, 'num_object_copies': 18916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7181, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2760}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:05.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6949", 'reported_seq': 7031, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.629503+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:11:02.629503+0000', 'last_peered': '2024-09-15T03:11:02.629503+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:11:02.629503+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6949, 'log_dups_size': 0, 'ondisk_log_size': 6949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36673138, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 17908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1991, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6947, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7413", 'reported_seq': 7513, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:03.188521+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:11:03.188521+0000', 'last_peered': '2024-09-15T03:11:03.188521+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:11:03.188521+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7413, 'log_dups_size': 0, 'ondisk_log_size': 7413, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39377745, 'num_objects': 4808, 'num_object_clones': 0, 'num_object_copies': 19232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4808, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7412, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2714}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6986", 'reported_seq': 9158, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.744809+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:11:02.744809+0000', 'last_peered': '2024-09-15T03:11:02.744809+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:11:02.744809+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 6986, 'log_dups_size': 0, 'ondisk_log_size': 6986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38024818, 'num_objects': 4642, 'num_object_clones': 0, 'num_object_copies': 18568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 892, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6984, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1055, 'num_bytes_recovered': 8560640, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3750}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7309", 'reported_seq': 7395, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.880171+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:11:02.880171+0000', 'last_peered': '2024-09-15T03:11:02.880171+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:11:02.880171+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7309, 'log_dups_size': 0, 'ondisk_log_size': 7309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39089778, 'num_objects': 4772, 'num_object_clones': 0, 'num_object_copies': 19088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7307, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2679}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7285", 'reported_seq': 7351, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.889353+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:11:02.889353+0000', 'last_peered': '2024-09-15T03:11:02.889353+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:11:02.889353+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7285, 'log_dups_size': 0, 'ondisk_log_size': 7285, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38608896, 'num_objects': 4713, 'num_object_clones': 0, 'num_object_copies': 18852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7285, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2667}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:11:05.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7247", 'reported_seq': 7334, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.712564+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:11:02.712564+0000', 'last_peered': '2024-09-15T03:11:02.712564+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:11:02.712564+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7247, 'log_dups_size': 0, 'ondisk_log_size': 7247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38590066, 'num_objects': 4711, 'num_object_clones': 0, 'num_object_copies': 18844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4711, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7245, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2669}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7291", 'reported_seq': 7376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.719664+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:11:02.719664+0000', 'last_peered': '2024-09-15T03:11:02.719664+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:11:02.719664+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7291, 'log_dups_size': 0, 'ondisk_log_size': 7291, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39270002, 'num_objects': 4794, 'num_object_clones': 0, 'num_object_copies': 19176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7289, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2723}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7290", 'reported_seq': 7373, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.947320+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:11:02.947320+0000', 'last_peered': '2024-09-15T03:11:02.947320+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:11:02.947320+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7290, 'log_dups_size': 0, 'ondisk_log_size': 7290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38333668, 'num_objects': 4680, 'num_object_clones': 0, 'num_object_copies': 18720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7287, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2650}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7270", 'reported_seq': 7373, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:03.143059+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:11:03.143059+0000', 'last_peered': '2024-09-15T03:11:03.143059+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:11:03.143059+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7270, 'log_dups_size': 0, 'ondisk_log_size': 7270, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38483570, 'num_objects': 4698, 'num_object_clones': 0, 'num_object_copies': 18792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7268, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2649}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7232", 'reported_seq': 7310, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.974078+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:11:02.974078+0000', 'last_peered': '2024-09-15T03:11:02.974078+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:11:02.974078+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7232, 'log_dups_size': 0, 'ondisk_log_size': 7232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38097323, 'num_objects': 4651, 'num_object_clones': 0, 'num_object_copies': 18604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7229, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2582}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7340", 'reported_seq': 7431, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:03.196844+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:11:03.196844+0000', 'last_peered': '2024-09-15T03:11:03.196844+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:11:03.196844+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7340, 'log_dups_size': 0, 'ondisk_log_size': 7340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39263033, 'num_objects': 4793, 'num_object_clones': 0, 'num_object_copies': 19172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7339, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:05.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:11:05.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7091", 'reported_seq': 7163, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.763853+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:11:02.763853+0000', 'last_peered': '2024-09-15T03:11:02.763853+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:11:02.763853+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7091, 'log_dups_size': 0, 'ondisk_log_size': 7091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37607026, 'num_objects': 4591, 'num_object_clones': 0, 'num_object_copies': 18364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7089, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2565}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:05.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:11:05.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7166", 'reported_seq': 7257, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:03.239431+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:11:03.239431+0000', 'last_peered': '2024-09-15T03:11:03.239431+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:11:03.239431+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7166, 'log_dups_size': 0, 'ondisk_log_size': 7166, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38639218, 'num_objects': 4717, 'num_object_clones': 0, 'num_object_copies': 18868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7164, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2733}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:05.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:11:05.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7273", 'reported_seq': 7360, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:02.646818+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:11:02.646818+0000', 'last_peered': '2024-09-15T03:11:02.646818+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:11:02.646818+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7273, 'log_dups_size': 0, 'ondisk_log_size': 7273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38794866, 'num_objects': 4736, 'num_object_clones': 0, 'num_object_copies': 18944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7271, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2665}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:05.411 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:11:07.200 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:07.200 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29482/300760 objects degraded (9.803%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":390,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":75190,"data_bytes":615914068,"bytes_used":2426216448,"bytes_avail":384120840192,"bytes_total":386547056640,"degraded_objects":29482,"degraded_total":300760,"degraded_ratio":0.098025003324910209,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":94168,"recovering_keys_per_sec":0,"num_objects_recovered":164,"num_bytes_recovered":1130496,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:11:07.237 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:11:07.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:08.731 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:08.768 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:08.797 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-15T03:11:06.792905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615914068,"num_objects":75190,"num_object_clones":0,"num_object_copies":300760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75190,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115517,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":24879104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115544,"ondisk_log_size":115544,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2369352,"kb_used_data":1157672,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375118008,"statfs":{"total":386547056640,"available":384120840192,"internally_reserved":0,"allocated":1185456128,"data_stored":1184790534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,5,12,4,8,7,1,0,0,0,1,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18685952,"num_objects":-2281,"num_object_clones":0,"num_object_copies":-9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004504"},"pg_stats":[{"pgid":"1.f","version":"33'7181","reported_seq":7239,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.828526+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:02.828526+0000","last_peered":"2024-09-15T03:11:02.828526+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:02.828526+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7181,"log_dups_size":0,"ondisk_log_size":7181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38739968,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7181,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2760}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6949","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.629503+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:02.629503+0000","last_peered":"2024-09-15T03:11:02.629503+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:02.629503+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6949,"log_dups_size":0,"ondisk_log_size":6949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36673138,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6947,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7413","reported_seq":7513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:03.188521+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:03.188521+0000","last_peered":"2024-09-15T03:11:03.188521+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:03.188521+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39377745,"num_objects":4808,"num_object_clones":0,"num_object_copies":19232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4808,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7412,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6986","reported_seq":9158,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.744809+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:02.744809+0000","last_peered":"2024-09-15T03:11:02.744809+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:02.744809+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38024818,"num_objects":4642,"num_object_clones":0,"num_object_copies":18568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":8560640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7309","reported_seq":7395,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.880171+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:02.880171+0000","last_peered":"2024-09-15T03:11:02.880171+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:02.880171+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7309,"log_dups_size":0,"ondisk_log_size":7309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39089778,"num_objects":4772,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7285","reported_seq":7351,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.889353+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:02.889353+0000","last_peered":"2024-09-15T03:11:02.889353+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:02.889353+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7285,"log_dups_size":0,"ondisk_log_size":7285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38608896,"num_objects":4713,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7285,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7247","reported_seq":7334,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.712564+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:02.712564+0000","last_peered":"2024-09-15T03:11:02.712564+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:02.712564+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7247,"log_dups_size":0,"ondisk_log_size":7247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38590066,"num_objects":4711,"num_object_clones":0,"num_object_copies":18844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7245,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2669}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7291","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.719664+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:02.719664+0000","last_peered":"2024-09-15T03:11:02.719664+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:02.719664+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7291,"log_dups_size":0,"ondisk_log_size":7291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39270002,"num_objects":4794,"num_object_clones":0,"num_object_copies":19176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7289,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2723}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7290","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.947320+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:02.947320+0000","last_peered":"2024-09-15T03:11:02.947320+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:02.947320+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7290,"log_dups_size":0,"ondisk_log_size":7290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38333668,"num_objects":4680,"num_object_clones":0,"num_object_copies":18720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7287,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7270","reported_seq":7373,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:03.143059+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:03.143059+0000","last_peered":"2024-09-15T03:11:03.143059+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:03.143059+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7270,"log_dups_size":0,"ondisk_log_size":7270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38483570,"num_objects":4698,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7268,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7232","reported_seq":7310,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.974078+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:02.974078+0000","last_peered":"2024-09-15T03:11:02.974078+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:02.974078+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38097323,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7229,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2582}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7221","reported_seq":11260,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:03.270903+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:03.270903+0000","last_peered":"2024-09-15T03:11:03.270903+0000","last_clean":"2024-09-15T03:11:03.270903+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:03.270903+0000","last_undegraded":"2024-09-15T03:11:03.270903+0000","last_fullsized":"2024-09-15T03:11:03.270903+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7221,"log_dups_size":0,"ondisk_log_size":7221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38320953,"num_objects":4678,"num_object_clones":0,"num_object_copies":18712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7220,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7340","reported_seq":7431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:03.196844+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:03.196844+0000","last_peered":"2024-09-15T03:11:03.196844+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:03.196844+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7340,"log_dups_size":0,"ondisk_log_size":7340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39263033,"num_objects":4793,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7339,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7091","reported_seq":7163,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.763853+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:02.763853+0000","last_peered":"2024-09-15T03:11:02.763853+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:02.763853+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7091,"log_dups_size":0,"ondisk_log_size":7091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37607026,"num_objects":4591,"num_object_clones":0,"num_object_copies":18364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7089,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2565}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7166","reported_seq":7257,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:03.239431+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:03.239431+0000","last_peered":"2024-09-15T03:11:03.239431+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:03.239431+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7166,"log_dups_size":0,"ondisk_log_size":7166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38639218,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2733}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7273","reported_seq":7360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:02.646818+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:02.646818+0000","last_peered":"2024-09-15T03:11:02.646818+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:02.646818+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7273,"log_dups_size":0,"ondisk_log_size":7273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38794866,"num_objects":4736,"num_object_clones":0,"num_object_copies":18944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7271,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":615914068,"num_objects":75190,"num_object_clones":0,"num_object_copies":300760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75190,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115517,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3202,"num_bytes_recovered":24879104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1183838208,"data_stored":1183838208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115544,"ondisk_log_size":115544,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":611472,"kb_used_data":301128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93760368,"statfs":{"total":96636764160,"available":96010616832,"internally_reserved":0,"allocated":308355072,"data_stored":308187887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,6,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":1.0309999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.30599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":2.7789999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":2.899}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598328,"kb_used_data":300976,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93773512,"statfs":{"total":96636764160,"available":96024076288,"internally_reserved":0,"allocated":308199424,"data_stored":308035385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,0,1,2,0,0,0,0,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.32200000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.371},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.40699999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.439},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.39500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556264,"kb_used_data":254240,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93815576,"statfs":{"total":96636764160,"available":96067149824,"internally_reserved":0,"allocated":260341760,"data_stored":260174575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.20999999999999999},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.37}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":20.055}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.317}]}]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603288,"kb_used_data":301328,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93768552,"statfs":{"total":96636764160,"available":96018997248,"internally_reserved":0,"allocated":308559872,"data_stored":308392687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,4,1,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.19700000000000001},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.9500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":7.1100000000000003},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.34599999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":308133888,"data_stored":308133888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260055040,"data_stored":260055040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307716096,"data_stored":307716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307933184,"data_stored":307933184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:11.799 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:13.120 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:13.180 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:13.195 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-15T03:11:12.794381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606706260,"num_objects":74066,"num_object_clones":0,"num_object_copies":296264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116641,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3284,"num_bytes_recovered":25534464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116668,"ondisk_log_size":116668,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2364132,"kb_used_data":1144112,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375123228,"statfs":{"total":386547056640,"available":384126185472,"internally_reserved":0,"allocated":1171570688,"data_stored":1170905094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,3,5,1,5,4,11,8,0,0,0,1,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18612224,"num_objects":-2272,"num_object_clones":0,"num_object_copies":-9088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2272,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"1.f","version":"33'7259","reported_seq":7316,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.912481+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:07.912481+0000","last_peered":"2024-09-15T03:11:07.912481+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:07.912481+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7259,"log_dups_size":0,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7259,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7017","reported_seq":7099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.823952+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:07.823952+0000","last_peered":"2024-09-15T03:11:07.823952+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:07.823952+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7017,"log_dups_size":0,"ondisk_log_size":7017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36116082,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7015,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7492","reported_seq":7594,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:08.078160+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:08.078160+0000","last_peered":"2024-09-15T03:11:08.078160+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:08.078160+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7492,"log_dups_size":0,"ondisk_log_size":7492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38730577,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7491,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2636}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7043","reported_seq":9374,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.852558+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:07.852558+0000","last_peered":"2024-09-15T03:11:07.852558+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:07.852558+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37557874,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1135,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7393","reported_seq":7479,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.915680+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:07.915680+0000","last_peered":"2024-09-15T03:11:07.915680+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:07.915680+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38401650,"num_objects":4688,"num_object_clones":0,"num_object_copies":18752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7391,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2595}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7347","reported_seq":7413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.836104+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:07.836104+0000","last_peered":"2024-09-15T03:11:07.836104+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:07.836104+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7347,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7321","reported_seq":7408,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.776913+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:07.776913+0000","last_peered":"2024-09-15T03:11:07.776913+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:07.776913+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7321,"log_dups_size":0,"ondisk_log_size":7321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37983858,"num_objects":4637,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7319,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2595}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7353","reported_seq":7438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.679283+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:07.679283+0000","last_peered":"2024-09-15T03:11:07.679283+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:07.679283+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7353,"log_dups_size":0,"ondisk_log_size":7353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38762098,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7346","reported_seq":7429,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.992787+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:07.992787+0000","last_peered":"2024-09-15T03:11:07.992787+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:07.992787+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7346,"log_dups_size":0,"ondisk_log_size":7346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37874916,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7338","reported_seq":7441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:08.239808+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:08.239808+0000","last_peered":"2024-09-15T03:11:08.239808+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:08.239808+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926514,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7336,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7305","reported_seq":7383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.872410+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:07.872410+0000","last_peered":"2024-09-15T03:11:07.872410+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:07.872410+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7305,"log_dups_size":0,"ondisk_log_size":7305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37499307,"num_objects":4578,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7302,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7289","reported_seq":11328,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:08.141794+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:08.141794+0000","last_peered":"2024-09-15T03:11:08.141794+0000","last_clean":"2024-09-15T03:11:08.141794+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:08.141794+0000","last_undegraded":"2024-09-15T03:11:08.141794+0000","last_fullsized":"2024-09-15T03:11:08.141794+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7289,"log_dups_size":0,"ondisk_log_size":7289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37763897,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7288,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7416","reported_seq":7506,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:08.245637+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:08.245637+0000","last_peered":"2024-09-15T03:11:08.245637+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:08.245637+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7416,"log_dups_size":0,"ondisk_log_size":7416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38640441,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7415,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7171","reported_seq":7243,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.991500+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:07.991500+0000","last_peered":"2024-09-15T03:11:07.991500+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:07.991500+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7171,"log_dups_size":0,"ondisk_log_size":7171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36951666,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7169,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7238","reported_seq":7331,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:08.308904+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:08.308904+0000","last_peered":"2024-09-15T03:11:08.308904+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:08.308904+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38049394,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2662}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7340","reported_seq":7427,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.739369+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:07.739369+0000","last_peered":"2024-09-15T03:11:07.739369+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:07.739369+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7340,"log_dups_size":0,"ondisk_log_size":7340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38246002,"num_objects":4669,"num_object_clones":0,"num_object_copies":18676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606706260,"num_objects":74066,"num_object_clones":0,"num_object_copies":296264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116641,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3284,"num_bytes_recovered":25534464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1169719296,"data_stored":1169719296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116668,"ondisk_log_size":116668,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598584,"kb_used_data":296556,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93773256,"statfs":{"total":96636764160,"available":96023814144,"internally_reserved":0,"allocated":303673344,"data_stored":303506159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,0,1,1,2,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":8.2240000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.20599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":4.2690000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":5.798}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":5.6360000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":9.7430000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602292,"kb_used_data":296604,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93769548,"statfs":{"total":96636764160,"available":96020017152,"internally_reserved":0,"allocated":303722496,"data_stored":303558457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,1,2,0,6,4,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.217}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.46400000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560272,"kb_used_data":254088,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93811568,"statfs":{"total":96636764160,"available":96063045632,"internally_reserved":0,"allocated":260186112,"data_stored":260018927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.29899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.245},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":3.081}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":11.413},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.39600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602984,"kb_used_data":296864,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93768856,"statfs":{"total":96636764160,"available":96019308544,"internally_reserved":0,"allocated":303988736,"data_stored":303821551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.23499999999999999},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.307}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303472640,"data_stored":303472640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259854336,"data_stored":259854336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303054848,"data_stored":303054848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303337472,"data_stored":303337472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:13.197 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:14.508 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:14.542 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:14.589 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-15T03:11:12.794381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606706260,"num_objects":74066,"num_object_clones":0,"num_object_copies":296264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116641,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3284,"num_bytes_recovered":25534464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116668,"ondisk_log_size":116668,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2364132,"kb_used_data":1144112,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375123228,"statfs":{"total":386547056640,"available":384126185472,"internally_reserved":0,"allocated":1171570688,"data_stored":1170905094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,3,5,1,5,4,11,8,0,0,0,1,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18612224,"num_objects":-2272,"num_object_clones":0,"num_object_copies":-9088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2272,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"1.f","version":"33'7259","reported_seq":7316,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.912481+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:07.912481+0000","last_peered":"2024-09-15T03:11:07.912481+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:07.912481+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7259,"log_dups_size":0,"ondisk_log_size":7259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7259,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7017","reported_seq":7099,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.823952+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:07.823952+0000","last_peered":"2024-09-15T03:11:07.823952+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:07.823952+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7017,"log_dups_size":0,"ondisk_log_size":7017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36116082,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7015,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7492","reported_seq":7594,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:08.078160+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:08.078160+0000","last_peered":"2024-09-15T03:11:08.078160+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:08.078160+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7492,"log_dups_size":0,"ondisk_log_size":7492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38730577,"num_objects":4729,"num_object_clones":0,"num_object_copies":18916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4729,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7491,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2636}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7043","reported_seq":9374,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.852558+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:07.852558+0000","last_peered":"2024-09-15T03:11:07.852558+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:07.852558+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7043,"log_dups_size":0,"ondisk_log_size":7043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37557874,"num_objects":4585,"num_object_clones":0,"num_object_copies":18340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1135,"num_bytes_recovered":9216000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7393","reported_seq":7479,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.915680+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:07.915680+0000","last_peered":"2024-09-15T03:11:07.915680+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:07.915680+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38401650,"num_objects":4688,"num_object_clones":0,"num_object_copies":18752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7391,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2595}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7347","reported_seq":7413,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.836104+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:07.836104+0000","last_peered":"2024-09-15T03:11:07.836104+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:07.836104+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7347,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2605}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7321","reported_seq":7408,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.776913+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:07.776913+0000","last_peered":"2024-09-15T03:11:07.776913+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:07.776913+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7321,"log_dups_size":0,"ondisk_log_size":7321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37983858,"num_objects":4637,"num_object_clones":0,"num_object_copies":18548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7319,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2595}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7353","reported_seq":7438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.679283+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:07.679283+0000","last_peered":"2024-09-15T03:11:07.679283+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:07.679283+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7353,"log_dups_size":0,"ondisk_log_size":7353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38762098,"num_objects":4732,"num_object_clones":0,"num_object_copies":18928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2661}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7346","reported_seq":7429,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.992787+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:07.992787+0000","last_peered":"2024-09-15T03:11:07.992787+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:07.992787+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7346,"log_dups_size":0,"ondisk_log_size":7346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37874916,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7343,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7338","reported_seq":7441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:08.239808+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:08.239808+0000","last_peered":"2024-09-15T03:11:08.239808+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:08.239808+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7338,"log_dups_size":0,"ondisk_log_size":7338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926514,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7336,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7305","reported_seq":7383,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.872410+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:07.872410+0000","last_peered":"2024-09-15T03:11:07.872410+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:07.872410+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7305,"log_dups_size":0,"ondisk_log_size":7305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37499307,"num_objects":4578,"num_object_clones":0,"num_object_copies":18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7302,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2509}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7289","reported_seq":11328,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:08.141794+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:08.141794+0000","last_peered":"2024-09-15T03:11:08.141794+0000","last_clean":"2024-09-15T03:11:08.141794+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:08.141794+0000","last_undegraded":"2024-09-15T03:11:08.141794+0000","last_fullsized":"2024-09-15T03:11:08.141794+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7289,"log_dups_size":0,"ondisk_log_size":7289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37763897,"num_objects":4610,"num_object_clones":0,"num_object_copies":18440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7288,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7416","reported_seq":7506,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:08.245637+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:08.245637+0000","last_peered":"2024-09-15T03:11:08.245637+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:08.245637+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7416,"log_dups_size":0,"ondisk_log_size":7416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38640441,"num_objects":4717,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7415,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2684}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7171","reported_seq":7243,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.991500+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:07.991500+0000","last_peered":"2024-09-15T03:11:07.991500+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:07.991500+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7171,"log_dups_size":0,"ondisk_log_size":7171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36951666,"num_objects":4511,"num_object_clones":0,"num_object_copies":18044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7169,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2485}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7238","reported_seq":7331,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:08.308904+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:08.308904+0000","last_peered":"2024-09-15T03:11:08.308904+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:08.308904+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38049394,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7236,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2662}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7340","reported_seq":7427,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:07.739369+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:07.739369+0000","last_peered":"2024-09-15T03:11:07.739369+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:07.739369+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7340,"log_dups_size":0,"ondisk_log_size":7340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38246002,"num_objects":4669,"num_object_clones":0,"num_object_copies":18676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7338,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606706260,"num_objects":74066,"num_object_clones":0,"num_object_copies":296264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74066,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116641,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3284,"num_bytes_recovered":25534464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1169719296,"data_stored":1169719296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116668,"ondisk_log_size":116668,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598584,"kb_used_data":296556,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93773256,"statfs":{"total":96636764160,"available":96023814144,"internally_reserved":0,"allocated":303673344,"data_stored":303506159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,0,1,1,2,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":8.2240000000000002},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.20599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":4.2690000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":5.798}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":5.6360000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":9.7430000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602292,"kb_used_data":296604,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93769548,"statfs":{"total":96636764160,"available":96020017152,"internally_reserved":0,"allocated":303722496,"data_stored":303558457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,1,2,0,6,4,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":0.19400000000000001},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.217}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.46400000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560272,"kb_used_data":254088,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93811568,"statfs":{"total":96636764160,"available":96063045632,"internally_reserved":0,"allocated":260186112,"data_stored":260018927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.3759999999999999,"15min":1.97},"min":{"1min":0.17799999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":11.047000000000001,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.956,"5min":1.6779999999999999,"15min":1.524},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":20.791,"5min":20.791,"15min":20.791},"last":0.29899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.5820000000000001,"15min":1.464},"min":{"1min":0.27000000000000002,"5min":0.219,"15min":0.219},"max":{"1min":12.962999999999999,"5min":12.962999999999999,"15min":12.962999999999999},"last":0.245},{"interface":"front","average":{"1min":3.1219999999999999,"5min":1.6930000000000001,"15min":1.7250000000000001},"min":{"1min":0.24299999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.305999999999999,"5min":14.305999999999999,"15min":14.305999999999999},"last":3.081}]},{"osd":3,"last update":"Sun Sep 15 03:10:08 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":1.5149999999999999,"15min":1.611},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":10.680999999999999,"5min":13.414999999999999,"15min":13.414999999999999},"last":11.413},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.3720000000000001,"15min":1.272},"min":{"1min":0.26500000000000001,"5min":0.251,"15min":0.251},"max":{"1min":7.8600000000000003,"5min":13.973000000000001,"15min":13.973000000000001},"last":0.39600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602984,"kb_used_data":296864,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93768856,"statfs":{"total":96636764160,"available":96019308544,"internally_reserved":0,"allocated":303988736,"data_stored":303821551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.23499999999999999},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.307}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303472640,"data_stored":303472640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259854336,"data_stored":259854336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303054848,"data_stored":303054848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303337472,"data_stored":303337472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:14.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:11:14.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7259", 'reported_seq': 7316, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.912481+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:11:07.912481+0000', 'last_peered': '2024-09-15T03:11:07.912481+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:11:07.912481+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7259, 'log_dups_size': 0, 'ondisk_log_size': 7259, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38100992, 'num_objects': 4651, 'num_object_clones': 0, 'num_object_copies': 18604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7259, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2682}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:14.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:11:14.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7017", 'reported_seq': 7099, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.823952+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:11:07.823952+0000', 'last_peered': '2024-09-15T03:11:07.823952+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:11:07.823952+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7017, 'log_dups_size': 0, 'ondisk_log_size': 7017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36116082, 'num_objects': 4409, 'num_object_clones': 0, 'num_object_copies': 17636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1991, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7015, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:14.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:11:14.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7492", 'reported_seq': 7594, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:08.078160+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:11:08.078160+0000', 'last_peered': '2024-09-15T03:11:08.078160+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:11:08.078160+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7492, 'log_dups_size': 0, 'ondisk_log_size': 7492, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38730577, 'num_objects': 4729, 'num_object_clones': 0, 'num_object_copies': 18916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2094, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4729, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7491, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2636}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:14.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:11:14.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7043", 'reported_seq': 9374, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.852558+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:11:07.852558+0000', 'last_peered': '2024-09-15T03:11:07.852558+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:11:07.852558+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7043, 'log_dups_size': 0, 'ondisk_log_size': 7043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37557874, 'num_objects': 4585, 'num_object_clones': 0, 'num_object_copies': 18340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 812, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7041, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1135, 'num_bytes_recovered': 9216000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7393", 'reported_seq': 7479, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.915680+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:11:07.915680+0000', 'last_peered': '2024-09-15T03:11:07.915680+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:11:07.915680+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7393, 'log_dups_size': 0, 'ondisk_log_size': 7393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38401650, 'num_objects': 4688, 'num_object_clones': 0, 'num_object_copies': 18752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7391, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2595}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7347", 'reported_seq': 7413, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.836104+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:11:07.836104+0000', 'last_peered': '2024-09-15T03:11:07.836104+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:11:07.836104+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7347, 'log_dups_size': 0, 'ondisk_log_size': 7347, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38100992, 'num_objects': 4651, 'num_object_clones': 0, 'num_object_copies': 18604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7347, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2605}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7321", 'reported_seq': 7408, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.776913+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:11:07.776913+0000', 'last_peered': '2024-09-15T03:11:07.776913+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:11:07.776913+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7321, 'log_dups_size': 0, 'ondisk_log_size': 7321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37983858, 'num_objects': 4637, 'num_object_clones': 0, 'num_object_copies': 18548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7319, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2595}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7353", 'reported_seq': 7438, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.679283+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:11:07.679283+0000', 'last_peered': '2024-09-15T03:11:07.679283+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:11:07.679283+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7353, 'log_dups_size': 0, 'ondisk_log_size': 7353, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38762098, 'num_objects': 4732, 'num_object_clones': 0, 'num_object_copies': 18928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7351, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:11:14.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7346", 'reported_seq': 7429, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.992787+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:11:07.992787+0000', 'last_peered': '2024-09-15T03:11:07.992787+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:11:07.992787+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7346, 'log_dups_size': 0, 'ondisk_log_size': 7346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37874916, 'num_objects': 4624, 'num_object_clones': 0, 'num_object_copies': 18496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7343, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2594}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:14.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:11:14.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7338", 'reported_seq': 7441, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:08.239808+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:11:08.239808+0000', 'last_peered': '2024-09-15T03:11:08.239808+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:11:08.239808+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7338, 'log_dups_size': 0, 'ondisk_log_size': 7338, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37926514, 'num_objects': 4630, 'num_object_clones': 0, 'num_object_copies': 18520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7336, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2581}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:14.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:11:14.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7305", 'reported_seq': 7383, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.872410+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:11:07.872410+0000', 'last_peered': '2024-09-15T03:11:07.872410+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:11:07.872410+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7305, 'log_dups_size': 0, 'ondisk_log_size': 7305, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37499307, 'num_objects': 4578, 'num_object_clones': 0, 'num_object_copies': 18312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7302, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2509}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:14.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:11:14.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7416", 'reported_seq': 7506, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:08.245637+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:11:08.245637+0000', 'last_peered': '2024-09-15T03:11:08.245637+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:11:08.245637+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7416, 'log_dups_size': 0, 'ondisk_log_size': 7416, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38640441, 'num_objects': 4717, 'num_object_clones': 0, 'num_object_copies': 18868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7415, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2684}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:14.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:11:14.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7171", 'reported_seq': 7243, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.991500+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:11:07.991500+0000', 'last_peered': '2024-09-15T03:11:07.991500+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:11:07.991500+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7171, 'log_dups_size': 0, 'ondisk_log_size': 7171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36951666, 'num_objects': 4511, 'num_object_clones': 0, 'num_object_copies': 18044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7169, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2485}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:14.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:11:14.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7238", 'reported_seq': 7331, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:08.308904+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:11:08.308904+0000', 'last_peered': '2024-09-15T03:11:08.308904+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:11:08.308904+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7238, 'log_dups_size': 0, 'ondisk_log_size': 7238, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38049394, 'num_objects': 4645, 'num_object_clones': 0, 'num_object_copies': 18580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7236, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2662}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:14.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:11:14.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7340", 'reported_seq': 7427, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:07.739369+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:11:07.739369+0000', 'last_peered': '2024-09-15T03:11:07.739369+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:11:07.739369+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7340, 'log_dups_size': 0, 'ondisk_log_size': 7340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38246002, 'num_objects': 4669, 'num_object_clones': 0, 'num_object_copies': 18676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2073, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7338, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2598}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:14.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:11:16.545 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:16.545 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29327/291732 objects degraded (10.053%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":399,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72933,"data_bytes":597424724,"bytes_used":2423660544,"bytes_avail":384123396096,"bytes_total":386547056640,"degraded_objects":29327,"degraded_total":291732,"degraded_ratio":0.10052719619376688,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":98279,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":1179648,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":256},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:11:16.571 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:11:16.571 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:18.330 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:18.333 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:18.353 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-15T03:11:16.795336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597424724,"num_objects":72933,"num_object_clones":0,"num_object_copies":291732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117774,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3357,"num_bytes_recovered":26058752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117801,"ondisk_log_size":117801,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2366856,"kb_used_data":1130184,"kb_used_omap":14,"kb_used_meta":1236593,"kb_avail":375120504,"statfs":{"total":386547056640,"available":384123396096,"internally_reserved":0,"allocated":1157308416,"data_stored":1156642822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266271259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,10,10,13,7,7,0,0,0,0,0,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18489344,"num_objects":-2257,"num_object_clones":0,"num_object_copies":-9028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2257,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003049"},"pg_stats":[{"pgid":"1.f","version":"33'7314","reported_seq":7372,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.944770+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:12.944770+0000","last_peered":"2024-09-15T03:11:12.944770+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:12.944770+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37650432,"num_objects":4596,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7314,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2627}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7076","reported_seq":7158,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.790317+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:12.790317+0000","last_peered":"2024-09-15T03:11:12.790317+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:12.790317+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7076,"log_dups_size":0,"ondisk_log_size":7076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35632754,"num_objects":4350,"num_object_clones":0,"num_object_copies":17400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7074,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7554","reported_seq":7656,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:13.285122+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:13.285122+0000","last_peered":"2024-09-15T03:11:13.285122+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:13.285122+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38222673,"num_objects":4667,"num_object_clones":0,"num_object_copies":18668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7553,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2574}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7114","reported_seq":9574,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.952023+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:12.952023+0000","last_peered":"2024-09-15T03:11:12.952023+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:12.952023+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7114,"log_dups_size":0,"ondisk_log_size":7114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36976242,"num_objects":4514,"num_object_clones":0,"num_object_copies":18056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7112,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1199,"num_bytes_recovered":9740288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7472","reported_seq":7558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.976358+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:12.976358+0000","last_peered":"2024-09-15T03:11:12.976358+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:12.976358+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7472,"log_dups_size":0,"ondisk_log_size":7472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37754482,"num_objects":4609,"num_object_clones":0,"num_object_copies":18436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2516}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7424","reported_seq":7488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.996379+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:12.996379+0000","last_peered":"2024-09-15T03:11:12.996379+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:12.996379+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7424,"log_dups_size":0,"ondisk_log_size":7424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37470208,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7397","reported_seq":7484,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.877246+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:12.877246+0000","last_peered":"2024-09-15T03:11:12.877246+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:12.877246+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37361266,"num_objects":4561,"num_object_clones":0,"num_object_copies":18244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2519}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7441","reported_seq":7534,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.764953+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:12.764953+0000","last_peered":"2024-09-15T03:11:12.764953+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:12.764953+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7441,"log_dups_size":0,"ondisk_log_size":7441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38041202,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7439,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7415","reported_seq":7497,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.638613+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:12.638613+0000","last_peered":"2024-09-15T03:11:12.638613+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:12.638613+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7415,"log_dups_size":0,"ondisk_log_size":7415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37309668,"num_objects":4555,"num_object_clones":0,"num_object_copies":18220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7414","reported_seq":7517,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:13.080279+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:13.080279+0000","last_peered":"2024-09-15T03:11:13.080279+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:13.080279+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7414,"log_dups_size":0,"ondisk_log_size":7414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37303922,"num_objects":4554,"num_object_clones":0,"num_object_copies":18216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2505}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7387","reported_seq":7465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:13.049366+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:13.049366+0000","last_peered":"2024-09-15T03:11:13.049366+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:13.049366+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7387,"log_dups_size":0,"ondisk_log_size":7387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36827563,"num_objects":4496,"num_object_clones":0,"num_object_copies":17984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7384,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2427}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7365","reported_seq":11404,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:13.262769+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:13.262769+0000","last_peered":"2024-09-15T03:11:13.262769+0000","last_clean":"2024-09-15T03:11:13.262769+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:13.262769+0000","last_undegraded":"2024-09-15T03:11:13.262769+0000","last_fullsized":"2024-09-15T03:11:13.262769+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7365,"log_dups_size":0,"ondisk_log_size":7365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37141305,"num_objects":4534,"num_object_clones":0,"num_object_copies":18136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7364,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7489","reported_seq":7580,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:13.224868+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:13.224868+0000","last_peered":"2024-09-15T03:11:13.224868+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:13.224868+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7489,"log_dups_size":0,"ondisk_log_size":7489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38042425,"num_objects":4644,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7232","reported_seq":7304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.990570+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:12.990570+0000","last_peered":"2024-09-15T03:11:12.990570+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:12.990570+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36451954,"num_objects":4450,"num_object_clones":0,"num_object_copies":17800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7230,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2424}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7297","reported_seq":7390,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.874078+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:12.874078+0000","last_peered":"2024-09-15T03:11:12.874078+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:12.874078+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7297,"log_dups_size":0,"ondisk_log_size":7297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37566066,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7295,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2603}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7410","reported_seq":7507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:12.895309+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:12.895309+0000","last_peered":"2024-09-15T03:11:12.895309+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:12.895309+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7410,"log_dups_size":0,"ondisk_log_size":7410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37672562,"num_objects":4599,"num_object_clones":0,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2533}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":597424724,"num_objects":72933,"num_object_clones":0,"num_object_copies":291732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117774,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3357,"num_bytes_recovered":26058752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155506176,"data_stored":1155506176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117801,"ondisk_log_size":117801,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598192,"kb_used_data":292000,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93773648,"statfs":{"total":96636764160,"available":96024215552,"internally_reserved":0,"allocated":299008000,"data_stored":298840815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,7,3,6,3,4,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.224},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.26400000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:10:11 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.867,"15min":1.7110000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":7.4560000000000004,"5min":13.31,"15min":13.31},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.2770000000000001,"5min":1.651,"15min":1.3360000000000001},"min":{"1min":0.246,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":13.343999999999999,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":0.53200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":601908,"kb_used_data":292076,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93769932,"statfs":{"total":96636764160,"available":96020410368,"internally_reserved":0,"allocated":299085824,"data_stored":298921785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,1,3,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":5.1959999999999997},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.313}]},{"osd":1,"last update":"Sun Sep 15 03:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.2150000000000001,"15min":1.107},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":2.6800000000000002,"5min":19.343,"15min":19.343},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.754,"5min":1.306,"15min":1.161},"min":{"1min":0.25900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.7170000000000001,"5min":23.311,"15min":23.311},"last":0.49199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.44800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564104,"kb_used_data":253760,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93807736,"statfs":{"total":96636764160,"available":96059121664,"internally_reserved":0,"allocated":259850240,"data_stored":259683055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":2.4369999999999998},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.23799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":2.4849999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":10.547000000000001},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":8.2590000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602652,"kb_used_data":292348,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93769188,"statfs":{"total":96636764160,"available":96019648512,"internally_reserved":0,"allocated":299364352,"data_stored":299197167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.9430000000000001,"5min":1.411,"15min":0.97199999999999998},"min":{"1min":0.188,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":18.594000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.6379999999999999},{"interface":"front","average":{"1min":4.0350000000000001,"5min":2.4359999999999999,"15min":1.825},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.266999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":9.4550000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":9.4130000000000003},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.254}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.33400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298811392,"data_stored":298811392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259530752,"data_stored":259530752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298475520,"data_stored":298475520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":298688512,"data_stored":298688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:21.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-15T03:11:20.796269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588175956,"num_objects":71804,"num_object_clones":0,"num_object_copies":287216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118903,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3421,"num_bytes_recovered":26583040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118930,"ondisk_log_size":118930,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2328996,"kb_used_data":1117312,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375158364,"statfs":{"total":386547056640,"available":384162164736,"internally_reserved":0,"allocated":1144127488,"data_stored":1143461894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,7,15,5,12,6,4,0,0,0,0,0,1,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18530304,"num_objects":-2262,"num_object_clones":0,"num_object_copies":-9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"1.f","version":"33'7369","reported_seq":7427,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.905459+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:17.905459+0000","last_peered":"2024-09-15T03:11:17.905459+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:17.905459+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7369,"log_dups_size":0,"ondisk_log_size":7369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37199872,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7150","reported_seq":7232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.702812+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:17.702812+0000","last_peered":"2024-09-15T03:11:17.702812+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:17.702812+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35026546,"num_objects":4276,"num_object_clones":0,"num_object_copies":17104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7635","reported_seq":7737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.354948+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:18.354948+0000","last_peered":"2024-09-15T03:11:18.354948+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:18.354948+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37559121,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7634,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2493}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7188","reported_seq":9776,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.935795+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:17.935795+0000","last_peered":"2024-09-15T03:11:17.935795+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:17.935795+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36370034,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7186,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":10264576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3756}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7538","reported_seq":7623,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.028885+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:18.028885+0000","last_peered":"2024-09-15T03:11:18.028885+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:18.028885+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7538,"log_dups_size":0,"ondisk_log_size":7538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37213810,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7536,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7498","reported_seq":7564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.080190+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:18.080190+0000","last_peered":"2024-09-15T03:11:18.080190+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:18.080190+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36864000,"num_objects":4500,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7461","reported_seq":7546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.838666+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:17.838666+0000","last_peered":"2024-09-15T03:11:17.838666+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:17.838666+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7461,"log_dups_size":0,"ondisk_log_size":7461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36836978,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7459,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7514","reported_seq":7607,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.849612+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:17.849612+0000","last_peered":"2024-09-15T03:11:17.849612+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:17.849612+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7514,"log_dups_size":0,"ondisk_log_size":7514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37443186,"num_objects":4571,"num_object_clones":0,"num_object_copies":18284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7512,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7475","reported_seq":7558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.994948+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:17.994948+0000","last_peered":"2024-09-15T03:11:17.994948+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:17.994948+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7475,"log_dups_size":0,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36818148,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7480","reported_seq":7583,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.152584+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:18.152584+0000","last_peered":"2024-09-15T03:11:18.152584+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:18.152584+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36763250,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7478,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7452","reported_seq":7530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.970807+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:17.970807+0000","last_peered":"2024-09-15T03:11:17.970807+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:17.970807+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7452,"log_dups_size":0,"ondisk_log_size":7452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36295083,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7449,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7429","reported_seq":11468,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:18.401235+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:18.401235+0000","last_peered":"2024-09-15T03:11:18.401235+0000","last_clean":"2024-09-15T03:11:18.401235+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:18.401235+0000","last_undegraded":"2024-09-15T03:11:18.401235+0000","last_fullsized":"2024-09-15T03:11:18.401235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36617017,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7556","reported_seq":7647,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.275090+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:18.275090+0000","last_peered":"2024-09-15T03:11:18.275090+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:18.275090+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7556,"log_dups_size":0,"ondisk_log_size":7556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37493561,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2544}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7310","reported_seq":7382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.936966+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:17.936966+0000","last_peered":"2024-09-15T03:11:17.936966+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:17.936966+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35812978,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7308,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7385","reported_seq":7478,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.247725+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:18.247725+0000","last_peered":"2024-09-15T03:11:18.247725+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:18.247725+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36845170,"num_objects":4498,"num_object_clones":0,"num_object_copies":17992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2515}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7490","reported_seq":7587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.737986+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:17.737986+0000","last_peered":"2024-09-15T03:11:17.737986+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:17.737986+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37017202,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":588175956,"num_objects":71804,"num_object_clones":0,"num_object_copies":287216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118903,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3421,"num_bytes_recovered":26583040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1141280768,"data_stored":1141280768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118930,"ondisk_log_size":118930,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598676,"kb_used_data":288332,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93773164,"statfs":{"total":96636764160,"available":96023719936,"internally_reserved":0,"allocated":295251968,"data_stored":295084783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,4,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":5.1340000000000003},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.28599999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.253},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":5.2009999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":5.3609999999999998},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":5.1749999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584988,"kb_used_data":287636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93786852,"statfs":{"total":96636764160,"available":96037736448,"internally_reserved":0,"allocated":294539264,"data_stored":294375225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,0,3,3,3,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":5.4530000000000003},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":12.606}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.20499999999999999},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.25700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.29899999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555448,"kb_used_data":253424,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93816392,"statfs":{"total":96636764160,"available":96067985408,"internally_reserved":0,"allocated":259506176,"data_stored":259338991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.217},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.98299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.248}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":1.2010000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589884,"kb_used_data":287920,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93781956,"statfs":{"total":96636764160,"available":96032722944,"internally_reserved":0,"allocated":294830080,"data_stored":294662895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,3,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.0009999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.96999999999999997},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.89700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":6.6289999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":6.6619999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":294195200,"data_stored":294195200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259198976,"data_stored":259198976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293826560,"data_stored":293826560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294060032,"data_stored":294060032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:22.427 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:23.767 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:23.775 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:23.823 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-15T03:11:22.796766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":588175956,"num_objects":71804,"num_object_clones":0,"num_object_copies":287216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118903,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3421,"num_bytes_recovered":26583040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118930,"ondisk_log_size":118930,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2328996,"kb_used_data":1117312,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375158364,"statfs":{"total":386547056640,"available":384162164736,"internally_reserved":0,"allocated":1144127488,"data_stored":1143461894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,7,15,5,12,6,4,0,0,0,0,0,1,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18530304,"num_objects":-2262,"num_object_clones":0,"num_object_copies":-9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002852"},"pg_stats":[{"pgid":"1.f","version":"33'7369","reported_seq":7427,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.905459+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:17.905459+0000","last_peered":"2024-09-15T03:11:17.905459+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:17.905459+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7369,"log_dups_size":0,"ondisk_log_size":7369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37199872,"num_objects":4541,"num_object_clones":0,"num_object_copies":18164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7150","reported_seq":7232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.702812+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:17.702812+0000","last_peered":"2024-09-15T03:11:17.702812+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:17.702812+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35026546,"num_objects":4276,"num_object_clones":0,"num_object_copies":17104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7635","reported_seq":7737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.354948+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:18.354948+0000","last_peered":"2024-09-15T03:11:18.354948+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:18.354948+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37559121,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7634,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2493}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7188","reported_seq":9776,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.935795+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:17.935795+0000","last_peered":"2024-09-15T03:11:17.935795+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:17.935795+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7188,"log_dups_size":0,"ondisk_log_size":7188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36370034,"num_objects":4440,"num_object_clones":0,"num_object_copies":17760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7186,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":10264576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3756}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7538","reported_seq":7623,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.028885+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:18.028885+0000","last_peered":"2024-09-15T03:11:18.028885+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:18.028885+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7538,"log_dups_size":0,"ondisk_log_size":7538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37213810,"num_objects":4543,"num_object_clones":0,"num_object_copies":18172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7536,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7498","reported_seq":7564,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.080190+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:18.080190+0000","last_peered":"2024-09-15T03:11:18.080190+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:18.080190+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36864000,"num_objects":4500,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7498,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2454}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7461","reported_seq":7546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.838666+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:17.838666+0000","last_peered":"2024-09-15T03:11:17.838666+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:17.838666+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7461,"log_dups_size":0,"ondisk_log_size":7461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36836978,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7459,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7514","reported_seq":7607,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.849612+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:17.849612+0000","last_peered":"2024-09-15T03:11:17.849612+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:17.849612+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7514,"log_dups_size":0,"ondisk_log_size":7514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37443186,"num_objects":4571,"num_object_clones":0,"num_object_copies":18284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7512,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7475","reported_seq":7558,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.994948+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:17.994948+0000","last_peered":"2024-09-15T03:11:17.994948+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:17.994948+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7475,"log_dups_size":0,"ondisk_log_size":7475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36818148,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7480","reported_seq":7583,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.152584+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:18.152584+0000","last_peered":"2024-09-15T03:11:18.152584+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:18.152584+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36763250,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7478,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7452","reported_seq":7530,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.970807+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:17.970807+0000","last_peered":"2024-09-15T03:11:17.970807+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:17.970807+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7452,"log_dups_size":0,"ondisk_log_size":7452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36295083,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7449,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7429","reported_seq":11468,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:18.401235+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:18.401235+0000","last_peered":"2024-09-15T03:11:18.401235+0000","last_clean":"2024-09-15T03:11:18.401235+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:18.401235+0000","last_undegraded":"2024-09-15T03:11:18.401235+0000","last_fullsized":"2024-09-15T03:11:18.401235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36617017,"num_objects":4470,"num_object_clones":0,"num_object_copies":17880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7428,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7556","reported_seq":7647,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.275090+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:18.275090+0000","last_peered":"2024-09-15T03:11:18.275090+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:18.275090+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7556,"log_dups_size":0,"ondisk_log_size":7556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37493561,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7555,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2544}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7310","reported_seq":7382,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.936966+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:17.936966+0000","last_peered":"2024-09-15T03:11:17.936966+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:17.936966+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7310,"log_dups_size":0,"ondisk_log_size":7310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35812978,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7308,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7385","reported_seq":7478,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:18.247725+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:18.247725+0000","last_peered":"2024-09-15T03:11:18.247725+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:18.247725+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36845170,"num_objects":4498,"num_object_clones":0,"num_object_copies":17992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2515}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7490","reported_seq":7587,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:17.737986+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:17.737986+0000","last_peered":"2024-09-15T03:11:17.737986+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:17.737986+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37017202,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2068,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":588175956,"num_objects":71804,"num_object_clones":0,"num_object_copies":287216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118903,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3421,"num_bytes_recovered":26583040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1141280768,"data_stored":1141280768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118930,"ondisk_log_size":118930,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598676,"kb_used_data":288332,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93773164,"statfs":{"total":96636764160,"available":96023719936,"internally_reserved":0,"allocated":295251968,"data_stored":295084783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,1,4,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":5.1340000000000003},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.28599999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.253},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":5.2009999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":5.3609999999999998},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":5.1749999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584988,"kb_used_data":287636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93786852,"statfs":{"total":96636764160,"available":96037736448,"internally_reserved":0,"allocated":294539264,"data_stored":294375225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,0,3,3,3,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":5.4530000000000003},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":12.606}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.20499999999999999},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.25700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.29899999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555448,"kb_used_data":253424,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93816392,"statfs":{"total":96636764160,"available":96067985408,"internally_reserved":0,"allocated":259506176,"data_stored":259338991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.217},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.98299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.248}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":1.2010000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589884,"kb_used_data":287920,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93781956,"statfs":{"total":96636764160,"available":96032722944,"internally_reserved":0,"allocated":294830080,"data_stored":294662895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,5,3,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.0009999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.96999999999999997},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.89700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":6.6289999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":6.6619999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":294195200,"data_stored":294195200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259198976,"data_stored":259198976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293826560,"data_stored":293826560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294060032,"data_stored":294060032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:23.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:11:23.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7369", 'reported_seq': 7427, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.905459+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:11:17.905459+0000', 'last_peered': '2024-09-15T03:11:17.905459+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:11:17.905459+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7369, 'log_dups_size': 0, 'ondisk_log_size': 7369, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37199872, 'num_objects': 4541, 'num_object_clones': 0, 'num_object_copies': 18164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7369, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2572}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:23.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:11:23.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7150", 'reported_seq': 7232, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.702812+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:11:17.702812+0000', 'last_peered': '2024-09-15T03:11:17.702812+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:11:17.702812+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7150, 'log_dups_size': 0, 'ondisk_log_size': 7150, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35026546, 'num_objects': 4276, 'num_object_clones': 0, 'num_object_copies': 17104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1991, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7148, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7635", 'reported_seq': 7737, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:18.354948+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:11:18.354948+0000', 'last_peered': '2024-09-15T03:11:18.354948+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:11:18.354948+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7635, 'log_dups_size': 0, 'ondisk_log_size': 7635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37559121, 'num_objects': 4586, 'num_object_clones': 0, 'num_object_copies': 18344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2094, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4586, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7634, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2493}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7188", 'reported_seq': 9776, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.935795+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:11:17.935795+0000', 'last_peered': '2024-09-15T03:11:17.935795+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:11:17.935795+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7188, 'log_dups_size': 0, 'ondisk_log_size': 7188, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36370034, 'num_objects': 4440, 'num_object_clones': 0, 'num_object_copies': 17760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 684, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7186, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1263, 'num_bytes_recovered': 10264576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3756}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7538", 'reported_seq': 7623, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:18.028885+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:11:18.028885+0000', 'last_peered': '2024-09-15T03:11:18.028885+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:11:18.028885+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7538, 'log_dups_size': 0, 'ondisk_log_size': 7538, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37213810, 'num_objects': 4543, 'num_object_clones': 0, 'num_object_copies': 18172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7536, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2450}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7498", 'reported_seq': 7564, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:18.080190+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:11:18.080190+0000', 'last_peered': '2024-09-15T03:11:18.080190+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:11:18.080190+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7498, 'log_dups_size': 0, 'ondisk_log_size': 7498, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36864000, 'num_objects': 4500, 'num_object_clones': 0, 'num_object_copies': 18000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7498, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2454}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:11:23.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7461", 'reported_seq': 7546, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.838666+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:11:17.838666+0000', 'last_peered': '2024-09-15T03:11:17.838666+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:11:17.838666+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7461, 'log_dups_size': 0, 'ondisk_log_size': 7461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36836978, 'num_objects': 4497, 'num_object_clones': 0, 'num_object_copies': 17988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7459, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2455}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7514", 'reported_seq': 7607, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.849612+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:11:17.849612+0000', 'last_peered': '2024-09-15T03:11:17.849612+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:11:17.849612+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7514, 'log_dups_size': 0, 'ondisk_log_size': 7514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37443186, 'num_objects': 4571, 'num_object_clones': 0, 'num_object_copies': 18284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2069, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7512, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7475", 'reported_seq': 7558, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.994948+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:11:17.994948+0000', 'last_peered': '2024-09-15T03:11:17.994948+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:11:17.994948+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7475, 'log_dups_size': 0, 'ondisk_log_size': 7475, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36818148, 'num_objects': 4495, 'num_object_clones': 0, 'num_object_copies': 17980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7472, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2465}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7480", 'reported_seq': 7583, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:18.152584+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:11:18.152584+0000', 'last_peered': '2024-09-15T03:11:18.152584+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:11:18.152584+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7480, 'log_dups_size': 0, 'ondisk_log_size': 7480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36763250, 'num_objects': 4488, 'num_object_clones': 0, 'num_object_copies': 17952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2051, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7478, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2439}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7452", 'reported_seq': 7530, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.970807+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:11:17.970807+0000', 'last_peered': '2024-09-15T03:11:17.970807+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:11:17.970807+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7452, 'log_dups_size': 0, 'ondisk_log_size': 7452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36295083, 'num_objects': 4431, 'num_object_clones': 0, 'num_object_copies': 17724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7449, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2362}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7556", 'reported_seq': 7647, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:18.275090+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:11:18.275090+0000', 'last_peered': '2024-09-15T03:11:18.275090+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:11:18.275090+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7556, 'log_dups_size': 0, 'ondisk_log_size': 7556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37493561, 'num_objects': 4577, 'num_object_clones': 0, 'num_object_copies': 18308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7555, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:23.826 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:11:23.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7310", 'reported_seq': 7382, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.936966+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:11:17.936966+0000', 'last_peered': '2024-09-15T03:11:17.936966+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:11:17.936966+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7310, 'log_dups_size': 0, 'ondisk_log_size': 7310, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35812978, 'num_objects': 4372, 'num_object_clones': 0, 'num_object_copies': 17488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7308, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:23.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:11:23.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7385", 'reported_seq': 7478, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:18.247725+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:11:18.247725+0000', 'last_peered': '2024-09-15T03:11:18.247725+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:11:18.247725+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7385, 'log_dups_size': 0, 'ondisk_log_size': 7385, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36845170, 'num_objects': 4498, 'num_object_clones': 0, 'num_object_copies': 17992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1985, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7383, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2515}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:23.827 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:11:23.827 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7490", 'reported_seq': 7587, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:17.737986+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:11:17.737986+0000', 'last_peered': '2024-09-15T03:11:17.737986+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:11:17.737986+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7490, 'log_dups_size': 0, 'ondisk_log_size': 7490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37017202, 'num_objects': 4519, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2068, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7488, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:23.827 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:11:25.999 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:25.999 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29182/282828 objects degraded (10.318%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":409,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70707,"data_bytes":579189332,"bytes_used":2371174400,"bytes_avail":384175882240,"bytes_total":386547056640,"degraded_objects":29182,"degraded_total":282828,"degraded_ratio":0.10317931746503176,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":130348,"recovering_keys_per_sec":0,"num_objects_recovered":218,"num_bytes_recovered":1564672,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":279},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:11:26.034 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:11:26.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:27.402 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:27.425 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:27.445 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-15T03:11:26.803612+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579189332,"num_objects":70707,"num_object_clones":0,"num_object_copies":282828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120000,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3502,"num_bytes_recovered":27099136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120027,"ondisk_log_size":120027,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2315600,"kb_used_data":1103896,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375171760,"statfs":{"total":386547056640,"available":384175882240,"internally_reserved":0,"allocated":1130389504,"data_stored":1129723910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,13,6,6,9,4,0,0,1,0,5,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18235392,"num_objects":-2226,"num_object_clones":0,"num_object_copies":-8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2226,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008793"},"pg_stats":[{"pgid":"1.f","version":"33'7431","reported_seq":7489,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:22.895202+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:22.895202+0000","last_peered":"2024-09-15T03:11:22.895202+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:22.895202+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7431,"log_dups_size":0,"ondisk_log_size":7431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36691968,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7431,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7218","reported_seq":7300,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:22.928623+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:22.928623+0000","last_peered":"2024-09-15T03:11:22.928623+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:22.928623+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7218,"log_dups_size":0,"ondisk_log_size":7218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34469490,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7216,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7702","reported_seq":7804,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.394265+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:23.394265+0000","last_peered":"2024-09-15T03:11:23.394265+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:23.394265+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7702,"log_dups_size":0,"ondisk_log_size":7702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37010257,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7701,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2426}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7268","reported_seq":9982,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.013825+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:23.013825+0000","last_peered":"2024-09-15T03:11:23.013825+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:23.013825+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7268,"log_dups_size":0,"ondisk_log_size":7268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35714674,"num_objects":4360,"num_object_clones":0,"num_object_copies":17440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":10780672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7602","reported_seq":7692,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.146007+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:23.146007+0000","last_peered":"2024-09-15T03:11:23.146007+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:23.146007+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36689522,"num_objects":4479,"num_object_clones":0,"num_object_copies":17916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7600,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7566","reported_seq":7632,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.035716+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:23.035716+0000","last_peered":"2024-09-15T03:11:23.035716+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:23.035716+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7566,"log_dups_size":0,"ondisk_log_size":7566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36306944,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2386}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7530","reported_seq":7623,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:22.837042+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:22.837042+0000","last_peered":"2024-09-15T03:11:22.837042+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:22.837042+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7530,"log_dups_size":0,"ondisk_log_size":7530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36271730,"num_objects":4428,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7528,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7597","reported_seq":7696,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.037915+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:23.037915+0000","last_peered":"2024-09-15T03:11:23.037915+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:23.037915+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7597,"log_dups_size":0,"ondisk_log_size":7597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36763250,"num_objects":4488,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7595,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2424}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7536","reported_seq":7619,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.053699+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:23.053699+0000","last_peered":"2024-09-15T03:11:23.053699+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:23.053699+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36318436,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7533,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2404}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7552","reported_seq":7655,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.372387+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:23.372387+0000","last_peered":"2024-09-15T03:11:23.372387+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:23.372387+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7552,"log_dups_size":0,"ondisk_log_size":7552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36173426,"num_objects":4416,"num_object_clones":0,"num_object_copies":17664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2367}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7497","reported_seq":7575,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:22.847874+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:22.847874+0000","last_peered":"2024-09-15T03:11:22.847874+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:22.847874+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35926443,"num_objects":4386,"num_object_clones":0,"num_object_copies":17544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7494,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2317}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7488","reported_seq":11527,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:23.558376+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:23.558376+0000","last_peered":"2024-09-15T03:11:23.558376+0000","last_clean":"2024-09-15T03:11:23.558376+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:23.558376+0000","last_undegraded":"2024-09-15T03:11:23.558376+0000","last_fullsized":"2024-09-15T03:11:23.558376+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7488,"log_dups_size":0,"ondisk_log_size":7488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36133689,"num_objects":4411,"num_object_clones":0,"num_object_copies":17644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7634","reported_seq":7725,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.356932+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:23.356932+0000","last_peered":"2024-09-15T03:11:23.356932+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:23.356932+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36854585,"num_objects":4499,"num_object_clones":0,"num_object_copies":17996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7633,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2466}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7389","reported_seq":7473,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:22.852372+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:22.852372+0000","last_peered":"2024-09-15T03:11:22.852372+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:22.852372+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7389,"log_dups_size":0,"ondisk_log_size":7389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35165810,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7387,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7463","reported_seq":7559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:23.380906+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:23.380906+0000","last_peered":"2024-09-15T03:11:23.380906+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:23.380906+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7463,"log_dups_size":0,"ondisk_log_size":7463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36206194,"num_objects":4420,"num_object_clones":0,"num_object_copies":17680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7461,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7554","reported_seq":7653,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:22.915397+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:22.915397+0000","last_peered":"2024-09-15T03:11:22.915397+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:22.915397+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36492914,"num_objects":4455,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7552,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2390}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":579189332,"num_objects":70707,"num_object_clones":0,"num_object_copies":282828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120000,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3502,"num_bytes_recovered":27099136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1127616512,"data_stored":1127616512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120027,"ondisk_log_size":120027,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581732,"kb_used_data":283868,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93790108,"statfs":{"total":96636764160,"available":96041070592,"internally_reserved":0,"allocated":290680832,"data_stored":290513647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,1,0,0,0,0,0,5],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":5.5190000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.27700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":40.710999999999999},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":5.5839999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":40.784999999999997},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":32.377000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584820,"kb_used_data":283284,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93787020,"statfs":{"total":96636764160,"available":96037908480,"internally_reserved":0,"allocated":290082816,"data_stored":289918777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,1,3,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":2.6549999999999998},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.42199999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.28699999999999998},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.437}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.38200000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559344,"kb_used_data":253160,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93812496,"statfs":{"total":96636764160,"available":96063995904,"internally_reserved":0,"allocated":259235840,"data_stored":259068655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":4.5759999999999996},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.27700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":6.9260000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.26200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589704,"kb_used_data":283584,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93782136,"statfs":{"total":96636764160,"available":96032907264,"internally_reserved":0,"allocated":290390016,"data_stored":290222831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,0,2,3,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.309},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.746}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.69299999999999995},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.32500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.503},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.47499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":289730560,"data_stored":289730560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258908160,"data_stored":258908160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289357824,"data_stored":289357824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":289619968,"data_stored":289619968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:30.448 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:31.901 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:31.936 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:31.961 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-15T03:11:30.804653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570210900,"num_objects":69611,"num_object_clones":0,"num_object_copies":278444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121096,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3575,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121123,"ondisk_log_size":121123,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2331148,"kb_used_data":1090340,"kb_used_omap":14,"kb_used_meta":1240753,"kb_avail":375156212,"statfs":{"total":386547056640,"available":384159961088,"internally_reserved":0,"allocated":1116508160,"data_stored":1115842566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1270531099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,6,15,5,6,8,0,0,3,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17965056,"num_objects":-2193,"num_object_clones":0,"num_object_copies":-8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008927"},"pg_stats":[{"pgid":"1.f","version":"33'7511","reported_seq":7569,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.167075+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:28.167075+0000","last_peered":"2024-09-15T03:11:28.167075+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:28.167075+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36036608,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7274","reported_seq":7356,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:27.894949+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:27.894949+0000","last_peered":"2024-09-15T03:11:27.894949+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:27.894949+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7274,"log_dups_size":0,"ondisk_log_size":7274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34010738,"num_objects":4152,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2163}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7767","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.431597+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:28.431597+0000","last_peered":"2024-09-15T03:11:28.431597+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:28.431597+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7767,"log_dups_size":0,"ondisk_log_size":7767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36477777,"num_objects":4454,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7766,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7350","reported_seq":10184,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.010543+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:28.010543+0000","last_peered":"2024-09-15T03:11:28.010543+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:28.010543+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7350,"log_dups_size":0,"ondisk_log_size":7350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35042930,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7348,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11272192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7669","reported_seq":7762,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.155575+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:28.155575+0000","last_peered":"2024-09-15T03:11:28.155575+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:28.155575+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36140658,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7627","reported_seq":7695,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.059210+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:28.059210+0000","last_peered":"2024-09-15T03:11:28.059210+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:28.059210+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35807232,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7585","reported_seq":7678,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:27.958234+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:27.958234+0000","last_peered":"2024-09-15T03:11:27.958234+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:27.958234+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35821170,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7676","reported_seq":7779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.004330+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:28.004330+0000","last_peered":"2024-09-15T03:11:28.004330+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:28.004330+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7676,"log_dups_size":0,"ondisk_log_size":7676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36116082,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7674,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7607","reported_seq":7690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.145371+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:28.145371+0000","last_peered":"2024-09-15T03:11:28.145371+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:28.145371+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35736804,"num_objects":4363,"num_object_clones":0,"num_object_copies":17452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7613","reported_seq":7724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.345204+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:28.345204+0000","last_peered":"2024-09-15T03:11:28.345204+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:28.345204+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7613,"log_dups_size":0,"ondisk_log_size":7613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35673714,"num_objects":4355,"num_object_clones":0,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7611,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7566","reported_seq":7644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:27.917416+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:27.917416+0000","last_peered":"2024-09-15T03:11:27.917416+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:27.917416+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7566,"log_dups_size":0,"ondisk_log_size":7566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35361195,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7564","reported_seq":11603,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:28.565728+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:28.565728+0000","last_peered":"2024-09-15T03:11:28.565728+0000","last_clean":"2024-09-15T03:11:28.565728+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:28.565728+0000","last_undegraded":"2024-09-15T03:11:28.565728+0000","last_fullsized":"2024-09-15T03:11:28.565728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7564,"log_dups_size":0,"ondisk_log_size":7564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35511097,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7700","reported_seq":7795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.358430+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:28.358430+0000","last_peered":"2024-09-15T03:11:28.358430+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:28.358430+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36313913,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7699,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7455","reported_seq":7543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.107266+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:28.107266+0000","last_peered":"2024-09-15T03:11:28.107266+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:28.107266+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34625138,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2209}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7525","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.433739+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:28.433739+0000","last_peered":"2024-09-15T03:11:28.433739+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:28.433739+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7525,"log_dups_size":0,"ondisk_log_size":7525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35698290,"num_objects":4358,"num_object_clones":0,"num_object_copies":17432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7523,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2377}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7634","reported_seq":7733,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.002807+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:28.002807+0000","last_peered":"2024-09-15T03:11:28.002807+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:28.002807+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35837554,"num_objects":4375,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7632,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":570210900,"num_objects":69611,"num_object_clones":0,"num_object_copies":278444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121096,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3575,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1113735168,"data_stored":1113735168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121123,"ondisk_log_size":121123,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585668,"kb_used_data":279484,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93786172,"statfs":{"total":96636764160,"available":96037040128,"internally_reserved":0,"allocated":286191616,"data_stored":286024431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,2,3,4,1,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.31}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":5.7869999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588748,"kb_used_data":278908,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93783092,"statfs":{"total":96636764160,"available":96033886208,"internally_reserved":0,"allocated":285601792,"data_stored":285437753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,0,1,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":7.9580000000000002},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.42999999999999999},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.48499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.59099999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567312,"kb_used_data":252808,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93804528,"statfs":{"total":96636764160,"available":96055836672,"internally_reserved":0,"allocated":258875392,"data_stored":258708207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.224},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.371}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":4.7770000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589420,"kb_used_data":279140,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93782420,"statfs":{"total":96636764160,"available":96033198080,"internally_reserved":0,"allocated":285839360,"data_stored":285672175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,6,0,1,6,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.6830000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.349}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":3.778},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":1.3089999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285237248,"data_stored":285237248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258539520,"data_stored":258539520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284848128,"data_stored":284848128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285110272,"data_stored":285110272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:31.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:33.311 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:33.358 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:33.390 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-15T03:11:32.805111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":570210900,"num_objects":69611,"num_object_clones":0,"num_object_copies":278444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121096,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3575,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121123,"ondisk_log_size":121123,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2331148,"kb_used_data":1090340,"kb_used_omap":14,"kb_used_meta":1240753,"kb_avail":375156212,"statfs":{"total":386547056640,"available":384159961088,"internally_reserved":0,"allocated":1116508160,"data_stored":1115842566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1270531099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,6,15,5,6,8,0,0,3,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17965056,"num_objects":-2193,"num_object_clones":0,"num_object_copies":-8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008841"},"pg_stats":[{"pgid":"1.f","version":"33'7511","reported_seq":7569,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.167075+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:28.167075+0000","last_peered":"2024-09-15T03:11:28.167075+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:28.167075+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36036608,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7511,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7274","reported_seq":7356,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:27.894949+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:27.894949+0000","last_peered":"2024-09-15T03:11:27.894949+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:27.894949+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7274,"log_dups_size":0,"ondisk_log_size":7274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34010738,"num_objects":4152,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7272,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2163}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7767","reported_seq":7871,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.431597+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:28.431597+0000","last_peered":"2024-09-15T03:11:28.431597+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:28.431597+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7767,"log_dups_size":0,"ondisk_log_size":7767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36477777,"num_objects":4454,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7766,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7350","reported_seq":10184,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.010543+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:28.010543+0000","last_peered":"2024-09-15T03:11:28.010543+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:28.010543+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7350,"log_dups_size":0,"ondisk_log_size":7350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35042930,"num_objects":4278,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7348,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":11272192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7669","reported_seq":7762,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.155575+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:28.155575+0000","last_peered":"2024-09-15T03:11:28.155575+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:28.155575+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7669,"log_dups_size":0,"ondisk_log_size":7669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36140658,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7667,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7627","reported_seq":7695,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.059210+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:28.059210+0000","last_peered":"2024-09-15T03:11:28.059210+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:28.059210+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7627,"log_dups_size":0,"ondisk_log_size":7627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35807232,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2326}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7585","reported_seq":7678,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:27.958234+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:27.958234+0000","last_peered":"2024-09-15T03:11:27.958234+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:27.958234+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7585,"log_dups_size":0,"ondisk_log_size":7585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35821170,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2334}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7676","reported_seq":7779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.004330+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:28.004330+0000","last_peered":"2024-09-15T03:11:28.004330+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:28.004330+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7676,"log_dups_size":0,"ondisk_log_size":7676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36116082,"num_objects":4409,"num_object_clones":0,"num_object_copies":17636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7674,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7607","reported_seq":7690,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.145371+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:28.145371+0000","last_peered":"2024-09-15T03:11:28.145371+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:28.145371+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7607,"log_dups_size":0,"ondisk_log_size":7607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35736804,"num_objects":4363,"num_object_clones":0,"num_object_copies":17452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2333}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7613","reported_seq":7724,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.345204+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:28.345204+0000","last_peered":"2024-09-15T03:11:28.345204+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:28.345204+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7613,"log_dups_size":0,"ondisk_log_size":7613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35673714,"num_objects":4355,"num_object_clones":0,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7611,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7566","reported_seq":7644,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:27.917416+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:27.917416+0000","last_peered":"2024-09-15T03:11:27.917416+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:27.917416+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7566,"log_dups_size":0,"ondisk_log_size":7566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35361195,"num_objects":4317,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7564","reported_seq":11603,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:28.565728+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:28.565728+0000","last_peered":"2024-09-15T03:11:28.565728+0000","last_clean":"2024-09-15T03:11:28.565728+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:28.565728+0000","last_undegraded":"2024-09-15T03:11:28.565728+0000","last_fullsized":"2024-09-15T03:11:28.565728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7564,"log_dups_size":0,"ondisk_log_size":7564,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35511097,"num_objects":4335,"num_object_clones":0,"num_object_copies":17340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7700","reported_seq":7795,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.358430+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:28.358430+0000","last_peered":"2024-09-15T03:11:28.358430+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:28.358430+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36313913,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7699,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7455","reported_seq":7543,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.107266+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:28.107266+0000","last_peered":"2024-09-15T03:11:28.107266+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:28.107266+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34625138,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2209}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7525","reported_seq":7622,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.433739+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:28.433739+0000","last_peered":"2024-09-15T03:11:28.433739+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:28.433739+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7525,"log_dups_size":0,"ondisk_log_size":7525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35698290,"num_objects":4358,"num_object_clones":0,"num_object_copies":17432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7523,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2377}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7634","reported_seq":7733,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:28.002807+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:28.002807+0000","last_peered":"2024-09-15T03:11:28.002807+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:28.002807+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35837554,"num_objects":4375,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7632,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":570210900,"num_objects":69611,"num_object_clones":0,"num_object_copies":278444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121096,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3575,"num_bytes_recovered":27590656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1113735168,"data_stored":1113735168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121123,"ondisk_log_size":121123,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585668,"kb_used_data":279484,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93786172,"statfs":{"total":96636764160,"available":96037040128,"internally_reserved":0,"allocated":286191616,"data_stored":286024431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,2,3,4,1,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.31}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":5.7869999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588748,"kb_used_data":278908,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93783092,"statfs":{"total":96636764160,"available":96033886208,"internally_reserved":0,"allocated":285601792,"data_stored":285437753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,0,1,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:27 2024","interfaces":[{"interface":"back","average":{"1min":2.5670000000000002,"5min":2.1419999999999999,"15min":1.1950000000000001},"min":{"1min":0.19800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":17.152999999999999,"5min":23.280000000000001,"15min":23.280000000000001},"last":7.9580000000000002},{"interface":"front","average":{"1min":2.7890000000000001,"5min":1.7909999999999999,"15min":1.1439999999999999},"min":{"1min":0.20300000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":19.329000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.42999999999999999},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.48499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:31 2024","interfaces":[{"interface":"back","average":{"1min":2.169,"5min":2.0059999999999998,"15min":1.0720000000000001},"min":{"1min":0.28999999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.242000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":3.4870000000000001,"5min":2.0960000000000001,"15min":1.2110000000000001},"min":{"1min":0.26200000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":22.710000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.59099999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567312,"kb_used_data":252808,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93804528,"statfs":{"total":96636764160,"available":96055836672,"internally_reserved":0,"allocated":258875392,"data_stored":258708207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.224},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.371}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":8.0700000000000003},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":4.7770000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589420,"kb_used_data":279140,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93782420,"statfs":{"total":96636764160,"available":96033198080,"internally_reserved":0,"allocated":285839360,"data_stored":285672175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,6,0,1,6,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.6830000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.349}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":3.778},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":1.3089999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":285237248,"data_stored":285237248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258539520,"data_stored":258539520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284848128,"data_stored":284848128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285110272,"data_stored":285110272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:33.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:11:33.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7511", 'reported_seq': 7569, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.167075+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:11:28.167075+0000', 'last_peered': '2024-09-15T03:11:28.167075+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:11:28.167075+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7511, 'log_dups_size': 0, 'ondisk_log_size': 7511, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36036608, 'num_objects': 4399, 'num_object_clones': 0, 'num_object_copies': 17596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1969, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7511, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2430}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:33.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:11:33.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7274", 'reported_seq': 7356, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:27.894949+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:11:27.894949+0000', 'last_peered': '2024-09-15T03:11:27.894949+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:11:27.894949+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7274, 'log_dups_size': 0, 'ondisk_log_size': 7274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34010738, 'num_objects': 4152, 'num_object_clones': 0, 'num_object_copies': 16608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1991, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7272, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2163}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:33.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:11:33.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7767", 'reported_seq': 7871, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.431597+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:11:28.431597+0000', 'last_peered': '2024-09-15T03:11:28.431597+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:11:28.431597+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7767, 'log_dups_size': 0, 'ondisk_log_size': 7767, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36477777, 'num_objects': 4454, 'num_object_clones': 0, 'num_object_copies': 17816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2093, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4454, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7766, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2362}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:33.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:11:33.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7350", 'reported_seq': 10184, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.010543+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:11:28.010543+0000', 'last_peered': '2024-09-15T03:11:28.010543+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:11:28.010543+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7350, 'log_dups_size': 0, 'ondisk_log_size': 7350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35042930, 'num_objects': 4278, 'num_object_clones': 0, 'num_object_copies': 17112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 561, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7348, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1386, 'num_bytes_recovered': 11272192, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:33.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:11:33.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7669", 'reported_seq': 7762, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.155575+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:11:28.155575+0000', 'last_peered': '2024-09-15T03:11:28.155575+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:11:28.155575+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7669, 'log_dups_size': 0, 'ondisk_log_size': 7669, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36140658, 'num_objects': 4412, 'num_object_clones': 0, 'num_object_copies': 17648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2091, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7667, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2323}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:33.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:11:33.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7627", 'reported_seq': 7695, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.059210+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:11:28.059210+0000', 'last_peered': '2024-09-15T03:11:28.059210+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:11:28.059210+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7627, 'log_dups_size': 0, 'ondisk_log_size': 7627, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35807232, 'num_objects': 4371, 'num_object_clones': 0, 'num_object_copies': 17484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7627, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2326}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7585", 'reported_seq': 7678, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:27.958234+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:11:27.958234+0000', 'last_peered': '2024-09-15T03:11:27.958234+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:11:27.958234+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7585, 'log_dups_size': 0, 'ondisk_log_size': 7585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35821170, 'num_objects': 4373, 'num_object_clones': 0, 'num_object_copies': 17492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2041, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7583, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2334}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7676", 'reported_seq': 7779, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.004330+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:11:28.004330+0000', 'last_peered': '2024-09-15T03:11:28.004330+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:11:28.004330+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7676, 'log_dups_size': 0, 'ondisk_log_size': 7676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36116082, 'num_objects': 4409, 'num_object_clones': 0, 'num_object_copies': 17636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2064, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7674, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2347}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7607", 'reported_seq': 7690, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.145371+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:11:28.145371+0000', 'last_peered': '2024-09-15T03:11:28.145371+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:11:28.145371+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7607, 'log_dups_size': 0, 'ondisk_log_size': 7607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35736804, 'num_objects': 4363, 'num_object_clones': 0, 'num_object_copies': 17452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7604, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2333}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7613", 'reported_seq': 7724, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.345204+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:11:28.345204+0000', 'last_peered': '2024-09-15T03:11:28.345204+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:11:28.345204+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7613, 'log_dups_size': 0, 'ondisk_log_size': 7613, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35673714, 'num_objects': 4355, 'num_object_clones': 0, 'num_object_copies': 17420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7611, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2310}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:33.393 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7566", 'reported_seq': 7644, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:27.917416+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:11:27.917416+0000', 'last_peered': '2024-09-15T03:11:27.917416+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:11:27.917416+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7566, 'log_dups_size': 0, 'ondisk_log_size': 7566, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35361195, 'num_objects': 4317, 'num_object_clones': 0, 'num_object_copies': 17268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7563, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2248}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7700", 'reported_seq': 7795, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.358430+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:11:28.358430+0000', 'last_peered': '2024-09-15T03:11:28.358430+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:11:28.358430+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7700, 'log_dups_size': 0, 'ondisk_log_size': 7700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36313913, 'num_objects': 4433, 'num_object_clones': 0, 'num_object_copies': 17732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2032, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7699, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7455", 'reported_seq': 7543, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.107266+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:11:28.107266+0000', 'last_peered': '2024-09-15T03:11:28.107266+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:11:28.107266+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7455, 'log_dups_size': 0, 'ondisk_log_size': 7455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34625138, 'num_objects': 4227, 'num_object_clones': 0, 'num_object_copies': 16908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2020, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7453, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7525", 'reported_seq': 7622, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.433739+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:11:28.433739+0000', 'last_peered': '2024-09-15T03:11:28.433739+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:11:28.433739+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7525, 'log_dups_size': 0, 'ondisk_log_size': 7525, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35698290, 'num_objects': 4358, 'num_object_clones': 0, 'num_object_copies': 17432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1983, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7523, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2377}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:11:33.394 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7634", 'reported_seq': 7733, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:28.002807+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:11:28.002807+0000', 'last_peered': '2024-09-15T03:11:28.002807+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:11:28.002807+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7634, 'log_dups_size': 0, 'ondisk_log_size': 7634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35837554, 'num_objects': 4375, 'num_object_clones': 0, 'num_object_copies': 17500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2067, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7632, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2310}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:33.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:11:35.177 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:35.177 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29015/274060 objects degraded (10.587%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":418,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68515,"data_bytes":561232468,"bytes_used":2343686144,"bytes_avail":384203370496,"bytes_total":386547056640,"degraded_objects":29015,"degraded_total":274060,"degraded_ratio":0.1058709771582865,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":143937,"recovering_keys_per_sec":0,"num_objects_recovered":248,"num_bytes_recovered":1728512,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":273},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:11:35.217 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:11:35.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:36.445 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:36.448 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:36.518 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-15T03:11:34.805572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":561232468,"num_objects":68515,"num_object_clones":0,"num_object_copies":274060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122192,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3669,"num_bytes_recovered":28311552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122219,"ondisk_log_size":122219,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2288756,"kb_used_data":1077060,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375198604,"statfs":{"total":386547056640,"available":384203370496,"internally_reserved":0,"allocated":1102909440,"data_stored":1102243846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,4,10,10,10,5,0,0,2,0,1,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26943488,"num_objects":-3289,"num_object_clones":0,"num_object_copies":-13156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3289,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":1728512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008806"},"pg_stats":[{"pgid":"1.f","version":"33'7580","reported_seq":7640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.256929+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:33.256929+0000","last_peered":"2024-09-15T03:11:33.256929+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:33.256929+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35471360,"num_objects":4330,"num_object_clones":0,"num_object_copies":17320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7580,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2362}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7352","reported_seq":7434,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.085214+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:33.085214+0000","last_peered":"2024-09-15T03:11:33.085214+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:33.085214+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7352,"log_dups_size":0,"ondisk_log_size":7352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371762,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7350,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7843","reported_seq":7947,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.454327+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:33.454327+0000","last_peered":"2024-09-15T03:11:33.454327+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:33.454327+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7843,"log_dups_size":0,"ondisk_log_size":7843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35855185,"num_objects":4378,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7842,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2286}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7415","reported_seq":10425,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.112791+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:33.112791+0000","last_peered":"2024-09-15T03:11:33.112791+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:33.112791+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7415,"log_dups_size":0,"ondisk_log_size":7415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34510450,"num_objects":4213,"num_object_clones":0,"num_object_copies":16852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7413,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1474,"num_bytes_recovered":11993088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7744","reported_seq":7838,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.144147+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:33.144147+0000","last_peered":"2024-09-15T03:11:33.144147+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:33.144147+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35526258,"num_objects":4337,"num_object_clones":0,"num_object_copies":17348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7742,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2248}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7697","reported_seq":7765,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.038982+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:33.038982+0000","last_peered":"2024-09-15T03:11:33.038982+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:33.038982+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35233792,"num_objects":4301,"num_object_clones":0,"num_object_copies":17204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7697,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2256}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7648","reported_seq":7745,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:32.742679+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:32.742679+0000","last_peered":"2024-09-15T03:11:32.742679+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:32.742679+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7648,"log_dups_size":0,"ondisk_log_size":7648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35305074,"num_objects":4310,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7646,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2273}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7749","reported_seq":7852,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.120289+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:33.120289+0000","last_peered":"2024-09-15T03:11:33.120289+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:33.120289+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7749,"log_dups_size":0,"ondisk_log_size":7749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35518066,"num_objects":4336,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7747,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2274}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7667","reported_seq":7750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.160587+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:33.160587+0000","last_peered":"2024-09-15T03:11:33.160587+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:33.160587+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7667,"log_dups_size":0,"ondisk_log_size":7667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35245284,"num_objects":4303,"num_object_clones":0,"num_object_copies":17212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7664,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2273}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7689","reported_seq":7800,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.522672+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:33.522672+0000","last_peered":"2024-09-15T03:11:33.522672+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:33.522672+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7689,"log_dups_size":0,"ondisk_log_size":7689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35051122,"num_objects":4279,"num_object_clones":0,"num_object_copies":17116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7687,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2234}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7638","reported_seq":7718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:32.860565+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:32.860565+0000","last_peered":"2024-09-15T03:11:32.860565+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:32.860565+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7638,"log_dups_size":0,"ondisk_log_size":7638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34771371,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7635,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7614","reported_seq":11653,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:33.677772+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:33.677772+0000","last_peered":"2024-09-15T03:11:33.677772+0000","last_clean":"2024-09-15T03:11:33.677772+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:33.677772+0000","last_undegraded":"2024-09-15T03:11:33.677772+0000","last_fullsized":"2024-09-15T03:11:33.677772+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7614,"log_dups_size":0,"ondisk_log_size":7614,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35101497,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7613,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7760","reported_seq":7855,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.587304+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:33.587304+0000","last_peered":"2024-09-15T03:11:33.587304+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:33.587304+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7760,"log_dups_size":0,"ondisk_log_size":7760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35822393,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7759,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2342}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7524","reported_seq":7616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.234932+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:33.234932+0000","last_peered":"2024-09-15T03:11:33.234932+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:33.234932+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059890,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7522,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2142}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7586","reported_seq":7683,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:33.427933+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:33.427933+0000","last_peered":"2024-09-15T03:11:33.427933+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:33.427933+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35198578,"num_objects":4297,"num_object_clones":0,"num_object_copies":17188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7584,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7713","reported_seq":7812,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:32.928381+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:32.928381+0000","last_peered":"2024-09-15T03:11:32.928381+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:32.928381+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35190386,"num_objects":4296,"num_object_clones":0,"num_object_copies":17184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":561232468,"num_objects":68515,"num_object_clones":0,"num_object_copies":274060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68515,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122192,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3669,"num_bytes_recovered":28311552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1100128256,"data_stored":1100128256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122219,"ondisk_log_size":122219,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585472,"kb_used_data":275128,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93786368,"statfs":{"total":96636764160,"available":96037240832,"internally_reserved":0,"allocated":281731072,"data_stored":281563887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,2,3,2,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.39000000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.58999999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":0.23300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571852,"kb_used_data":274484,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93799988,"statfs":{"total":96636764160,"available":96051187712,"internally_reserved":0,"allocated":281071616,"data_stored":280907577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,4,2,4,3,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.68700000000000006},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.71699999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.254}]}]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554704,"kb_used_data":252680,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93817136,"statfs":{"total":96636764160,"available":96068747264,"internally_reserved":0,"allocated":258744320,"data_stored":258577135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.72699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.60599999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.3490000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":2.2480000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576728,"kb_used_data":274768,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93795112,"statfs":{"total":96636764160,"available":96046194688,"internally_reserved":0,"allocated":281362432,"data_stored":281195247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,2,2,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":2.464},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":2.3780000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:10:33 2024","interfaces":[{"interface":"back","average":{"1min":3.891,"5min":2.681,"15min":1.5169999999999999},"min":{"1min":0.222,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":30.306000000000001,"5min":30.802,"15min":30.802},"last":0.28100000000000003},{"interface":"front","average":{"1min":2.774,"5min":1.5940000000000001,"15min":0.93799999999999994},"min":{"1min":0.25600000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":15.066000000000001,"5min":15.066000000000001,"15min":15.066000000000001},"last":0.36699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":2.581},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.32100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280723456,"data_stored":280723456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258433024,"data_stored":258433024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280342528,"data_stored":280342528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280629248,"data_stored":280629248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:39.521 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:40.785 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:40.799 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:40.813 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-15T03:11:38.808631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":552221268,"num_objects":67415,"num_object_clones":0,"num_object_copies":269660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123292,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3775,"num_bytes_recovered":29106176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123319,"ondisk_log_size":123319,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2279764,"kb_used_data":1063900,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375207596,"statfs":{"total":386547056640,"available":384212578304,"internally_reserved":0,"allocated":1089433600,"data_stored":1088768006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,10,8,17,9,6,0,0,0,0,1,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26968064,"num_objects":-3292,"num_object_clones":0,"num_object_copies":-13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3292,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":2007040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005019"},"pg_stats":[{"pgid":"1.f","version":"33'7652","reported_seq":7712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.212531+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:38.212531+0000","last_peered":"2024-09-15T03:11:38.212531+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:38.212531+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34881536,"num_objects":4258,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7652,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7421","reported_seq":7507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.068806+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:38.068806+0000","last_peered":"2024-09-15T03:11:38.068806+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:38.068806+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7421,"log_dups_size":0,"ondisk_log_size":7421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32806514,"num_objects":4005,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7419,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7903","reported_seq":8007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.610793+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:38.610793+0000","last_peered":"2024-09-15T03:11:38.610793+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:38.610793+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35363665,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7902,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2226}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7478","reported_seq":10682,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.138528+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:38.138528+0000","last_peered":"2024-09-15T03:11:38.138528+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:38.138528+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33994354,"num_objects":4150,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7476,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":12787712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7826","reported_seq":7920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.116746+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:38.116746+0000","last_peered":"2024-09-15T03:11:38.116746+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:38.116746+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7826,"log_dups_size":0,"ondisk_log_size":7826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34854514,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7771","reported_seq":7838,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.262141+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:38.262141+0000","last_peered":"2024-09-15T03:11:38.262141+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:38.262141+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34627584,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7771,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7707","reported_seq":7810,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:37.997732+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:37.997732+0000","last_peered":"2024-09-15T03:11:37.997732+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:37.997732+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34821746,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7705,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2217}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7816","reported_seq":7923,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.140673+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:38.140673+0000","last_peered":"2024-09-15T03:11:38.140673+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:38.140673+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7816,"log_dups_size":0,"ondisk_log_size":7816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34969202,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7814,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7735","reported_seq":7818,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.202160+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:38.202160+0000","last_peered":"2024-09-15T03:11:38.202160+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:38.202160+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7735,"log_dups_size":0,"ondisk_log_size":7735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34688228,"num_objects":4235,"num_object_clones":0,"num_object_copies":16940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7763","reported_seq":7873,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.407317+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:38.407317+0000","last_peered":"2024-09-15T03:11:38.407317+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:38.407317+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7763,"log_dups_size":0,"ondisk_log_size":7763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444914,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7761,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7698","reported_seq":7778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.140078+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:38.140078+0000","last_peered":"2024-09-15T03:11:38.140078+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:38.140078+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34279851,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7676","reported_seq":11715,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:38.720017+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:38.720017+0000","last_peered":"2024-09-15T03:11:38.720017+0000","last_clean":"2024-09-15T03:11:38.720017+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:38.720017+0000","last_undegraded":"2024-09-15T03:11:38.720017+0000","last_fullsized":"2024-09-15T03:11:38.720017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7676,"log_dups_size":0,"ondisk_log_size":7676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34593593,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7824","reported_seq":7921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.501956+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:38.501956+0000","last_peered":"2024-09-15T03:11:38.501956+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:38.501956+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35298105,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7823,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7608","reported_seq":7700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.210480+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:38.210480+0000","last_peered":"2024-09-15T03:11:38.210480+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:38.210480+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371762,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2058}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7653","reported_seq":7750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.447499+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:38.447499+0000","last_peered":"2024-09-15T03:11:38.447499+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:38.447499+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7653,"log_dups_size":0,"ondisk_log_size":7653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34649714,"num_objects":4230,"num_object_clones":0,"num_object_copies":16920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7651,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7788","reported_seq":7888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.059390+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:38.059390+0000","last_peered":"2024-09-15T03:11:38.059390+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:38.059390+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34575986,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7786,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":552221268,"num_objects":67415,"num_object_clones":0,"num_object_copies":269660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123292,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3775,"num_bytes_recovered":29106176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1086607360,"data_stored":1086607360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123319,"ondisk_log_size":123319,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568628,"kb_used_data":270764,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93803212,"statfs":{"total":96636764160,"available":96054489088,"internally_reserved":0,"allocated":277262336,"data_stored":277095151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,8,4,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.33300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":0.78900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571736,"kb_used_data":270204,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93800104,"statfs":{"total":96636764160,"available":96051306496,"internally_reserved":0,"allocated":276688896,"data_stored":276524857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,4,2,4,3,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.68700000000000006},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.71699999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.254}]}]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562916,"kb_used_data":252568,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93808924,"statfs":{"total":96636764160,"available":96060338176,"internally_reserved":0,"allocated":258629632,"data_stored":258462447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.72699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.60599999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.3490000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":2.2480000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576484,"kb_used_data":270364,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93795356,"statfs":{"total":96636764160,"available":96046444544,"internally_reserved":0,"allocated":276852736,"data_stored":276685551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,0,1,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.317},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.27000000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":8.2850000000000001},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":13.407}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.36299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":276271104,"data_stored":276271104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258322432,"data_stored":258322432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275861504,"data_stored":275861504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":276152320,"data_stored":276152320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:40.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:41.976 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:42.007 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:42.022 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-15T03:11:40.809189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":552221268,"num_objects":67415,"num_object_clones":0,"num_object_copies":269660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123292,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3775,"num_bytes_recovered":29106176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123319,"ondisk_log_size":123319,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2279764,"kb_used_data":1063900,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375207596,"statfs":{"total":386547056640,"available":384212578304,"internally_reserved":0,"allocated":1089433600,"data_stored":1088768006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,10,8,17,9,6,0,0,0,0,1,1,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17989632,"num_objects":-2196,"num_object_clones":0,"num_object_copies":-8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2196,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005110"},"pg_stats":[{"pgid":"1.f","version":"33'7652","reported_seq":7712,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.212531+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:38.212531+0000","last_peered":"2024-09-15T03:11:38.212531+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:38.212531+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34881536,"num_objects":4258,"num_object_clones":0,"num_object_copies":17032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7652,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7421","reported_seq":7507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.068806+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:38.068806+0000","last_peered":"2024-09-15T03:11:38.068806+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:38.068806+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7421,"log_dups_size":0,"ondisk_log_size":7421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32806514,"num_objects":4005,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7419,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7903","reported_seq":8007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.610793+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:38.610793+0000","last_peered":"2024-09-15T03:11:38.610793+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:38.610793+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7903,"log_dups_size":0,"ondisk_log_size":7903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35363665,"num_objects":4318,"num_object_clones":0,"num_object_copies":17272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7902,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2226}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7478","reported_seq":10682,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.138528+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:38.138528+0000","last_peered":"2024-09-15T03:11:38.138528+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:38.138528+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33994354,"num_objects":4150,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7476,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":12787712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7826","reported_seq":7920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.116746+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:38.116746+0000","last_peered":"2024-09-15T03:11:38.116746+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:38.116746+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7826,"log_dups_size":0,"ondisk_log_size":7826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34854514,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7824,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7771","reported_seq":7838,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.262141+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:38.262141+0000","last_peered":"2024-09-15T03:11:38.262141+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:38.262141+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34627584,"num_objects":4227,"num_object_clones":0,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7771,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7707","reported_seq":7810,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:37.997732+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:37.997732+0000","last_peered":"2024-09-15T03:11:37.997732+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:37.997732+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34821746,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7705,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2217}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7816","reported_seq":7923,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.140673+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:38.140673+0000","last_peered":"2024-09-15T03:11:38.140673+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:38.140673+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7816,"log_dups_size":0,"ondisk_log_size":7816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34969202,"num_objects":4269,"num_object_clones":0,"num_object_copies":17076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7814,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7735","reported_seq":7818,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.202160+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:38.202160+0000","last_peered":"2024-09-15T03:11:38.202160+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:38.202160+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7735,"log_dups_size":0,"ondisk_log_size":7735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34688228,"num_objects":4235,"num_object_clones":0,"num_object_copies":16940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7732,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7763","reported_seq":7873,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.407317+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:38.407317+0000","last_peered":"2024-09-15T03:11:38.407317+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:38.407317+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7763,"log_dups_size":0,"ondisk_log_size":7763,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34444914,"num_objects":4205,"num_object_clones":0,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7761,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2160}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7698","reported_seq":7778,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.140078+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:38.140078+0000","last_peered":"2024-09-15T03:11:38.140078+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:38.140078+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34279851,"num_objects":4185,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7676","reported_seq":11715,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:38.720017+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:38.720017+0000","last_peered":"2024-09-15T03:11:38.720017+0000","last_clean":"2024-09-15T03:11:38.720017+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:38.720017+0000","last_undegraded":"2024-09-15T03:11:38.720017+0000","last_fullsized":"2024-09-15T03:11:38.720017+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7676,"log_dups_size":0,"ondisk_log_size":7676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34593593,"num_objects":4223,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7675,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7824","reported_seq":7921,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.501956+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:38.501956+0000","last_peered":"2024-09-15T03:11:38.501956+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:38.501956+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7824,"log_dups_size":0,"ondisk_log_size":7824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35298105,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7823,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7608","reported_seq":7700,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.210480+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:38.210480+0000","last_peered":"2024-09-15T03:11:38.210480+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:38.210480+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371762,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2058}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7653","reported_seq":7750,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.447499+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:38.447499+0000","last_peered":"2024-09-15T03:11:38.447499+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:38.447499+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7653,"log_dups_size":0,"ondisk_log_size":7653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34649714,"num_objects":4230,"num_object_clones":0,"num_object_copies":16920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7651,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2249}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7788","reported_seq":7888,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:38.059390+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:38.059390+0000","last_peered":"2024-09-15T03:11:38.059390+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:38.059390+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7788,"log_dups_size":0,"ondisk_log_size":7788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34575986,"num_objects":4221,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7786,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":552221268,"num_objects":67415,"num_object_clones":0,"num_object_copies":269660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28909,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123292,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3775,"num_bytes_recovered":29106176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1086607360,"data_stored":1086607360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123319,"ondisk_log_size":123319,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":568628,"kb_used_data":270764,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93803212,"statfs":{"total":96636764160,"available":96054489088,"internally_reserved":0,"allocated":277262336,"data_stored":277095151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,8,4,2,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.887,"15min":1.758},"min":{"1min":0.187,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.819,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.98799999999999999,"5min":2.0110000000000001,"15min":2.048},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":4.0830000000000002,"5min":17.542000000000002,"15min":17.542000000000002},"last":0.33300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.58099999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":2.9169999999999998,"5min":2.266,"15min":1.1100000000000001},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":18.579999999999998,"5min":24.946000000000002,"15min":24.946000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.8149999999999999,"15min":1.2290000000000001},"min":{"1min":0.22600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":7.7709999999999999,"5min":21.398,"15min":21.398},"last":0.78900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571736,"kb_used_data":270204,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93800104,"statfs":{"total":96636764160,"available":96051306496,"internally_reserved":0,"allocated":276688896,"data_stored":276524857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,4,2,4,3,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.45700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.68700000000000006},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.71699999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.254}]}]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562916,"kb_used_data":252568,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93808924,"statfs":{"total":96636764160,"available":96060338176,"internally_reserved":0,"allocated":258629632,"data_stored":258462447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.72699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.60599999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.3490000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":2.2480000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576484,"kb_used_data":270364,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93795356,"statfs":{"total":96636764160,"available":96046444544,"internally_reserved":0,"allocated":276852736,"data_stored":276685551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,0,1,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.317},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.27000000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":8.2850000000000001},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":13.407}]},{"osd":3,"last update":"Sun Sep 15 03:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":1.6799999999999999,"15min":1},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1099999999999999,"5min":20.948,"15min":20.948},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.228,"5min":1.492,"15min":0.81000000000000005},"min":{"1min":0.26100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.8129999999999997,"5min":11.6,"15min":11.6},"last":0.36299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":276271104,"data_stored":276271104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258322432,"data_stored":258322432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275861504,"data_stored":275861504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":276152320,"data_stored":276152320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:42.023 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:11:42.023 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7652", 'reported_seq': 7712, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.212531+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:11:38.212531+0000', 'last_peered': '2024-09-15T03:11:38.212531+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:11:38.212531+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7652, 'log_dups_size': 0, 'ondisk_log_size': 7652, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34881536, 'num_objects': 4258, 'num_object_clones': 0, 'num_object_copies': 17032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7652, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2290}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7421", 'reported_seq': 7507, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.068806+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:11:38.068806+0000', 'last_peered': '2024-09-15T03:11:38.068806+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:11:38.068806+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7421, 'log_dups_size': 0, 'ondisk_log_size': 7421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32806514, 'num_objects': 4005, 'num_object_clones': 0, 'num_object_copies': 16020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7419, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2018}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7903", 'reported_seq': 8007, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.610793+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:11:38.610793+0000', 'last_peered': '2024-09-15T03:11:38.610793+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:11:38.610793+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7903, 'log_dups_size': 0, 'ondisk_log_size': 7903, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35363665, 'num_objects': 4318, 'num_object_clones': 0, 'num_object_copies': 17272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2093, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4318, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7902, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2226}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7478", 'reported_seq': 10682, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.138528+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:11:38.138528+0000', 'last_peered': '2024-09-15T03:11:38.138528+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:11:38.138528+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7478, 'log_dups_size': 0, 'ondisk_log_size': 7478, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33994354, 'num_objects': 4150, 'num_object_clones': 0, 'num_object_copies': 16600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 376, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4150, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7476, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1571, 'num_bytes_recovered': 12787712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3774}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7826", 'reported_seq': 7920, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.116746+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:11:38.116746+0000', 'last_peered': '2024-09-15T03:11:38.116746+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:11:38.116746+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7826, 'log_dups_size': 0, 'ondisk_log_size': 7826, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34854514, 'num_objects': 4255, 'num_object_clones': 0, 'num_object_copies': 17020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2091, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7824, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2166}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:42.024 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7771", 'reported_seq': 7838, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.262141+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:11:38.262141+0000', 'last_peered': '2024-09-15T03:11:38.262141+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:11:38.262141+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7771, 'log_dups_size': 0, 'ondisk_log_size': 7771, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34627584, 'num_objects': 4227, 'num_object_clones': 0, 'num_object_copies': 16908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7771, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2182}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7707", 'reported_seq': 7810, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:37.997732+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:11:37.997732+0000', 'last_peered': '2024-09-15T03:11:37.997732+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:11:37.997732+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7707, 'log_dups_size': 0, 'ondisk_log_size': 7707, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34821746, 'num_objects': 4251, 'num_object_clones': 0, 'num_object_copies': 17004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2036, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7705, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2217}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7816", 'reported_seq': 7923, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.140673+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:11:38.140673+0000', 'last_peered': '2024-09-15T03:11:38.140673+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:11:38.140673+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7816, 'log_dups_size': 0, 'ondisk_log_size': 7816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34969202, 'num_objects': 4269, 'num_object_clones': 0, 'num_object_copies': 17076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7814, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7735", 'reported_seq': 7818, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.202160+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:11:38.202160+0000', 'last_peered': '2024-09-15T03:11:38.202160+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:11:38.202160+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7735, 'log_dups_size': 0, 'ondisk_log_size': 7735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34688228, 'num_objects': 4235, 'num_object_clones': 0, 'num_object_copies': 16940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7732, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2205}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7763", 'reported_seq': 7873, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.407317+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:11:38.407317+0000', 'last_peered': '2024-09-15T03:11:38.407317+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:11:38.407317+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7763, 'log_dups_size': 0, 'ondisk_log_size': 7763, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34444914, 'num_objects': 4205, 'num_object_clones': 0, 'num_object_copies': 16820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2047, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7761, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2160}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:11:42.025 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7698", 'reported_seq': 7778, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.140078+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:11:38.140078+0000', 'last_peered': '2024-09-15T03:11:38.140078+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:11:38.140078+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7698, 'log_dups_size': 0, 'ondisk_log_size': 7698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34279851, 'num_objects': 4185, 'num_object_clones': 0, 'num_object_copies': 16740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2071, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7695, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2117}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:11:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7824", 'reported_seq': 7921, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.501956+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:11:38.501956+0000', 'last_peered': '2024-09-15T03:11:38.501956+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:11:38.501956+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7824, 'log_dups_size': 0, 'ondisk_log_size': 7824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35298105, 'num_objects': 4309, 'num_object_clones': 0, 'num_object_copies': 17236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2031, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7823, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:11:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7608", 'reported_seq': 7700, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.210480+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:11:38.210480+0000', 'last_peered': '2024-09-15T03:11:38.210480+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:11:38.210480+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7608, 'log_dups_size': 0, 'ondisk_log_size': 7608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33371762, 'num_objects': 4074, 'num_object_clones': 0, 'num_object_copies': 16296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2018, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7606, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:11:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7653", 'reported_seq': 7750, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.447499+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:11:38.447499+0000', 'last_peered': '2024-09-15T03:11:38.447499+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:11:38.447499+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7653, 'log_dups_size': 0, 'ondisk_log_size': 7653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34649714, 'num_objects': 4230, 'num_object_clones': 0, 'num_object_copies': 16920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1983, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7651, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2249}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:11:42.026 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7788", 'reported_seq': 7888, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:38.059390+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:11:38.059390+0000', 'last_peered': '2024-09-15T03:11:38.059390+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:11:38.059390+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7788, 'log_dups_size': 0, 'ondisk_log_size': 7788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34575986, 'num_objects': 4221, 'num_object_clones': 0, 'num_object_copies': 16884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7786, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:42.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:11:44.042 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:44.042 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28909/269660 objects degraded (10.721%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":427,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67415,"data_bytes":552221268,"bytes_used":2334478336,"bytes_avail":384212578304,"bytes_total":386547056640,"degraded_objects":28909,"degraded_total":269660,"degraded_ratio":0.10720536972483868,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":126199,"recovering_keys_per_sec":0,"num_objects_recovered":200,"num_bytes_recovered":1515520,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":182},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:11:44.099 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:11:44.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:45.535 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:45.547 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:45.563 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-15T03:11:44.814067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543472212,"num_objects":66347,"num_object_clones":0,"num_object_copies":265388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124360,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3864,"num_bytes_recovered":29745152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124387,"ondisk_log_size":124387,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2274248,"kb_used_data":1050084,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375213112,"statfs":{"total":386547056640,"available":384218226688,"internally_reserved":0,"allocated":1075286016,"data_stored":1074620422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,4,10,7,3,9,0,0,5,0,1,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26738688,"num_objects":-3264,"num_object_clones":0,"num_object_copies":-13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":289,"num_bytes_recovered":2154496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008956"},"pg_stats":[{"pgid":"1.f","version":"33'7707","reported_seq":7767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.394847+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:43.394847+0000","last_peered":"2024-09-15T03:11:43.394847+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:43.394847+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34430976,"num_objects":4203,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7707,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7483","reported_seq":7573,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.200899+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:43.200899+0000","last_peered":"2024-09-15T03:11:43.200899+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:43.200899+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7483,"log_dups_size":0,"ondisk_log_size":7483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32298610,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7481,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7972","reported_seq":8076,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.563378+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:43.563378+0000","last_peered":"2024-09-15T03:11:43.563378+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:43.563378+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7972,"log_dups_size":0,"ondisk_log_size":7972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34798417,"num_objects":4249,"num_object_clones":0,"num_object_copies":16996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7971,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2157}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7554","reported_seq":10913,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.302790+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:43.302790+0000","last_peered":"2024-09-15T03:11:43.302790+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:43.302790+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371762,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7552,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":13426688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3776}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7905","reported_seq":7999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.148568+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:43.148568+0000","last_peered":"2024-09-15T03:11:43.148568+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:43.148568+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7905,"log_dups_size":0,"ondisk_log_size":7905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34207346,"num_objects":4176,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7903,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2087}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7832","reported_seq":7900,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.210534+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:43.210534+0000","last_peered":"2024-09-15T03:11:43.210534+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:43.210534+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7832,"log_dups_size":0,"ondisk_log_size":7832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34127872,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7832,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2121}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7795","reported_seq":7904,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.266417+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:43.266417+0000","last_peered":"2024-09-15T03:11:43.266417+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:43.266417+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34100850,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7793,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2132}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7891","reported_seq":8006,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.179201+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:43.179201+0000","last_peered":"2024-09-15T03:11:43.179201+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:43.179201+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7891,"log_dups_size":0,"ondisk_log_size":7891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34354802,"num_objects":4194,"num_object_clones":0,"num_object_copies":16776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7889,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7786","reported_seq":7869,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.403855+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:43.403855+0000","last_peered":"2024-09-15T03:11:43.403855+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:43.403855+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7786,"log_dups_size":0,"ondisk_log_size":7786,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34270436,"num_objects":4184,"num_object_clones":0,"num_object_copies":16736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7783,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2154}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7818","reported_seq":7929,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.434092+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:43.434092+0000","last_peered":"2024-09-15T03:11:43.434092+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:43.434092+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7818,"log_dups_size":0,"ondisk_log_size":7818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33994354,"num_objects":4150,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2047,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7816,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2105}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7778","reported_seq":7858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.389235+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:43.389235+0000","last_peered":"2024-09-15T03:11:43.389235+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:43.389235+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7778,"log_dups_size":0,"ondisk_log_size":7778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33624491,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2071,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7775,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7739","reported_seq":11778,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:43.776223+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:43.776223+0000","last_peered":"2024-09-15T03:11:43.776223+0000","last_clean":"2024-09-15T03:11:43.776223+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:43.776223+0000","last_undegraded":"2024-09-15T03:11:43.776223+0000","last_fullsized":"2024-09-15T03:11:43.776223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7739,"log_dups_size":0,"ondisk_log_size":7739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34077497,"num_objects":4160,"num_object_clones":0,"num_object_copies":16640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7738,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7893","reported_seq":7990,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.575257+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:43.575257+0000","last_peered":"2024-09-15T03:11:43.575257+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:43.575257+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7893,"log_dups_size":0,"ondisk_log_size":7893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34732857,"num_objects":4240,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7892,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2210}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7670","reported_seq":7762,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.321834+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:43.321834+0000","last_peered":"2024-09-15T03:11:43.321834+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:43.321834+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32863858,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7668,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7710","reported_seq":7806,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.467616+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:43.467616+0000","last_peered":"2024-09-15T03:11:43.467616+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:43.467616+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34182770,"num_objects":4173,"num_object_clones":0,"num_object_copies":16692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7708,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2192}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7854","reported_seq":7958,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:43.280775+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:43.280775+0000","last_peered":"2024-09-15T03:11:43.280775+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:43.280775+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7854,"log_dups_size":0,"ondisk_log_size":7854,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34035314,"num_objects":4155,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7852,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":543472212,"num_objects":66347,"num_object_clones":0,"num_object_copies":265388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66347,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124360,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3864,"num_bytes_recovered":29745152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073135616,"data_stored":1073135616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124387,"ondisk_log_size":124387,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571904,"kb_used_data":265716,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93799936,"statfs":{"total":96636764160,"available":96051134464,"internally_reserved":0,"allocated":272093184,"data_stored":271925999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1,2,5,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":4.9530000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":1.639},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.24099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":1.6779999999999999},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":1.502}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575748,"kb_used_data":265916,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93796092,"statfs":{"total":96636764160,"available":96047198208,"internally_reserved":0,"allocated":272297984,"data_stored":272133945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.22900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":2.7709999999999999},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":2.8090000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":2.8540000000000001},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":2.8700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550156,"kb_used_data":252292,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93821684,"statfs":{"total":96636764160,"available":96073404416,"internally_reserved":0,"allocated":258347008,"data_stored":258179823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.20100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":1.7170000000000001},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":6.4669999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":1.746}]}]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576440,"kb_used_data":266160,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93795400,"statfs":{"total":96636764160,"available":96046489600,"internally_reserved":0,"allocated":272547840,"data_stored":272380655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,7,1,0,1,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":5.4020000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":5.4820000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.33800000000000002},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.255},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.41399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271818752,"data_stored":271818752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":258052096,"data_stored":258052096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":271536128,"data_stored":271536128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271728640,"data_stored":271728640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:48.565 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:50.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:50.218 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:50.231 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-15T03:11:48.815107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534452820,"num_objects":65246,"num_object_clones":0,"num_object_copies":260984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125461,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3962,"num_bytes_recovered":30490624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125488,"ondisk_log_size":125488,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2248616,"kb_used_data":1036936,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375238744,"statfs":{"total":386547056640,"available":384244473856,"internally_reserved":0,"allocated":1061822464,"data_stored":1061156870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,7,5,7,8,7,7,0,0,0,0,2,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26779648,"num_objects":-3269,"num_object_clones":0,"num_object_copies":-13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3269,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006954"},"pg_stats":[{"pgid":"1.f","version":"33'7771","reported_seq":7831,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.302567+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:48.302567+0000","last_peered":"2024-09-15T03:11:48.302567+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:48.302567+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33906688,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7771,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7559","reported_seq":7649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.260530+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:48.260530+0000","last_peered":"2024-09-15T03:11:48.260530+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:48.260530+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7559,"log_dups_size":0,"ondisk_log_size":7559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31676018,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7557,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8050","reported_seq":8154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.600009+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:48.600009+0000","last_peered":"2024-09-15T03:11:48.600009+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:48.600009+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8050,"log_dups_size":0,"ondisk_log_size":8050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34159441,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8049,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2079}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7608","reported_seq":11150,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.414523+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:48.414523+0000","last_peered":"2024-09-15T03:11:48.414523+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:48.414523+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32929394,"num_objects":4020,"num_object_clones":0,"num_object_copies":16080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1740,"num_bytes_recovered":14172160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7982","reported_seq":8080,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.384929+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:48.384929+0000","last_peered":"2024-09-15T03:11:48.384929+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:48.384929+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7982,"log_dups_size":0,"ondisk_log_size":7982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33576562,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7980,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2012}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7897","reported_seq":7965,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.282852+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:48.282852+0000","last_peered":"2024-09-15T03:11:48.282852+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:48.282852+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7897,"log_dups_size":0,"ondisk_log_size":7897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33595392,"num_objects":4101,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7897,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7871","reported_seq":7980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.401564+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:48.401564+0000","last_peered":"2024-09-15T03:11:48.401564+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:48.401564+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7871,"log_dups_size":0,"ondisk_log_size":7871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33478258,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2056}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7959","reported_seq":8074,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.417795+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:48.417795+0000","last_peered":"2024-09-15T03:11:48.417795+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:48.417795+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33797746,"num_objects":4126,"num_object_clones":0,"num_object_copies":16504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2070}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7856","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.376463+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:48.376463+0000","last_peered":"2024-09-15T03:11:48.376463+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:48.376463+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33696996,"num_objects":4114,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7853,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2085}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7883","reported_seq":7996,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.501690+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:48.501690+0000","last_peered":"2024-09-15T03:11:48.501690+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:48.501690+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7883,"log_dups_size":0,"ondisk_log_size":7883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461874,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7881,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2041}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7844","reported_seq":7926,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.413224+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:48.413224+0000","last_peered":"2024-09-15T03:11:48.413224+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:48.413224+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7844,"log_dups_size":0,"ondisk_log_size":7844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083819,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7799","reported_seq":11838,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:48.668465+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:48.668465+0000","last_peered":"2024-09-15T03:11:48.668465+0000","last_clean":"2024-09-15T03:11:48.668465+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:48.668465+0000","last_undegraded":"2024-09-15T03:11:48.668465+0000","last_fullsized":"2024-09-15T03:11:48.668465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33585977,"num_objects":4100,"num_object_clones":0,"num_object_copies":16400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7967","reported_seq":8064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.555745+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:48.555745+0000","last_peered":"2024-09-15T03:11:48.555745+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:48.555745+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7967,"log_dups_size":0,"ondisk_log_size":7967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34126649,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7966,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2137}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7737","reported_seq":7831,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.459033+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:48.459033+0000","last_peered":"2024-09-15T03:11:48.459033+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:48.459033+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7737,"log_dups_size":0,"ondisk_log_size":7737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32314994,"num_objects":3945,"num_object_clones":0,"num_object_copies":15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7735,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7780","reported_seq":7877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.597294+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:48.597294+0000","last_peered":"2024-09-15T03:11:48.597294+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:48.597294+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7780,"log_dups_size":0,"ondisk_log_size":7780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33609330,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7925","reported_seq":8030,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.272534+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:48.272534+0000","last_peered":"2024-09-15T03:11:48.272534+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:48.272534+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7925,"log_dups_size":0,"ondisk_log_size":7925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33453682,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7923,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534452820,"num_objects":65246,"num_object_clones":0,"num_object_copies":260984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125461,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3962,"num_bytes_recovered":30490624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1059450880,"data_stored":1059450880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125488,"ondisk_log_size":125488,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571708,"kb_used_data":261364,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93800132,"statfs":{"total":96636764160,"available":96051335168,"internally_reserved":0,"allocated":267636736,"data_stored":267469551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,0,1,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":5.0780000000000003},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.311},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":5.0359999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558844,"kb_used_data":261492,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93812996,"statfs":{"total":96636764160,"available":96064507904,"internally_reserved":0,"allocated":267767808,"data_stored":267603769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,0,2,7,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.184},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.68999999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.41099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.437},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.66100000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554216,"kb_used_data":252192,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93817624,"statfs":{"total":96636764160,"available":96069246976,"internally_reserved":0,"allocated":258244608,"data_stored":258077423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,2,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.55300000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.20200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.0409999999999999},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":8.8480000000000008}]}]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563848,"kb_used_data":261888,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93807992,"statfs":{"total":96636764160,"available":96059383808,"internally_reserved":0,"allocated":268173312,"data_stored":268006127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,4,2,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.42799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.216},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":6.0880000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.34399999999999997},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267235328,"data_stored":267235328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257966080,"data_stored":257966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267046912,"data_stored":267046912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":267202560,"data_stored":267202560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:50.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:51.388 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:51.410 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:51.414 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-15T03:11:50.815629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534452820,"num_objects":65246,"num_object_clones":0,"num_object_copies":260984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125461,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3962,"num_bytes_recovered":30490624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125488,"ondisk_log_size":125488,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2248616,"kb_used_data":1036936,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375238744,"statfs":{"total":386547056640,"available":384244473856,"internally_reserved":0,"allocated":1061822464,"data_stored":1061156870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,7,5,7,8,7,7,0,0,0,0,2,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17768448,"num_objects":-2169,"num_object_clones":0,"num_object_copies":-8676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006998"},"pg_stats":[{"pgid":"1.f","version":"33'7771","reported_seq":7831,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.302567+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:48.302567+0000","last_peered":"2024-09-15T03:11:48.302567+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:48.302567+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7771,"log_dups_size":0,"ondisk_log_size":7771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33906688,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1968,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7771,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2171}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7559","reported_seq":7649,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.260530+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:48.260530+0000","last_peered":"2024-09-15T03:11:48.260530+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:48.260530+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7559,"log_dups_size":0,"ondisk_log_size":7559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31676018,"num_objects":3867,"num_object_clones":0,"num_object_copies":15468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1987,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7557,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8050","reported_seq":8154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.600009+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:48.600009+0000","last_peered":"2024-09-15T03:11:48.600009+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:48.600009+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8050,"log_dups_size":0,"ondisk_log_size":8050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34159441,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8049,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2079}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7608","reported_seq":11150,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.414523+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:48.414523+0000","last_peered":"2024-09-15T03:11:48.414523+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:48.414523+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7608,"log_dups_size":0,"ondisk_log_size":7608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32929394,"num_objects":4020,"num_object_clones":0,"num_object_copies":16080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7606,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1740,"num_bytes_recovered":14172160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7982","reported_seq":8080,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.384929+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:48.384929+0000","last_peered":"2024-09-15T03:11:48.384929+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:48.384929+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7982,"log_dups_size":0,"ondisk_log_size":7982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33576562,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7980,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2012}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7897","reported_seq":7965,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.282852+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:48.282852+0000","last_peered":"2024-09-15T03:11:48.282852+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:48.282852+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7897,"log_dups_size":0,"ondisk_log_size":7897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33595392,"num_objects":4101,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7897,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7871","reported_seq":7980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.401564+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:48.401564+0000","last_peered":"2024-09-15T03:11:48.401564+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:48.401564+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7871,"log_dups_size":0,"ondisk_log_size":7871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33478258,"num_objects":4087,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2056}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7959","reported_seq":8074,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.417795+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:48.417795+0000","last_peered":"2024-09-15T03:11:48.417795+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:48.417795+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33797746,"num_objects":4126,"num_object_clones":0,"num_object_copies":16504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2070}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7856","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.376463+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:48.376463+0000","last_peered":"2024-09-15T03:11:48.376463+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:48.376463+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33696996,"num_objects":4114,"num_object_clones":0,"num_object_copies":16456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2032,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7853,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2085}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7883","reported_seq":7996,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.501690+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:48.501690+0000","last_peered":"2024-09-15T03:11:48.501690+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:48.501690+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7883,"log_dups_size":0,"ondisk_log_size":7883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461874,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7881,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2041}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7844","reported_seq":7926,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.413224+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:48.413224+0000","last_peered":"2024-09-15T03:11:48.413224+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:48.413224+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7844,"log_dups_size":0,"ondisk_log_size":7844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33083819,"num_objects":4039,"num_object_clones":0,"num_object_copies":16156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2070,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7799","reported_seq":11838,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:48.668465+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:48.668465+0000","last_peered":"2024-09-15T03:11:48.668465+0000","last_clean":"2024-09-15T03:11:48.668465+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:48.668465+0000","last_undegraded":"2024-09-15T03:11:48.668465+0000","last_fullsized":"2024-09-15T03:11:48.668465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33585977,"num_objects":4100,"num_object_clones":0,"num_object_copies":16400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7798,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'7967","reported_seq":8064,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.555745+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:48.555745+0000","last_peered":"2024-09-15T03:11:48.555745+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:48.555745+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7967,"log_dups_size":0,"ondisk_log_size":7967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34126649,"num_objects":4166,"num_object_clones":0,"num_object_copies":16664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7966,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2137}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7737","reported_seq":7831,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.459033+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:48.459033+0000","last_peered":"2024-09-15T03:11:48.459033+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:48.459033+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7737,"log_dups_size":0,"ondisk_log_size":7737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32314994,"num_objects":3945,"num_object_clones":0,"num_object_copies":15780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7735,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1930}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7780","reported_seq":7877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.597294+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:48.597294+0000","last_peered":"2024-09-15T03:11:48.597294+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:48.597294+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7780,"log_dups_size":0,"ondisk_log_size":7780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33609330,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2122}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7925","reported_seq":8030,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:48.272534+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:48.272534+0000","last_peered":"2024-09-15T03:11:48.272534+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:48.272534+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7925,"log_dups_size":0,"ondisk_log_size":7925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33453682,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2064,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7923,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534452820,"num_objects":65246,"num_object_clones":0,"num_object_copies":260984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28722,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125461,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3962,"num_bytes_recovered":30490624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1059450880,"data_stored":1059450880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125488,"ondisk_log_size":125488,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571708,"kb_used_data":261364,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93800132,"statfs":{"total":96636764160,"available":96051335168,"internally_reserved":0,"allocated":267636736,"data_stored":267469551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,0,1,0,0,0,0,0,0,0,0,1],"upper_bound":65536},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":5.0780000000000003},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.311},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.27300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":5.0359999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558844,"kb_used_data":261492,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93812996,"statfs":{"total":96636764160,"available":96064507904,"internally_reserved":0,"allocated":267767808,"data_stored":267603769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,0,2,7,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.184},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.68999999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.41099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.437},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.66100000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554216,"kb_used_data":252192,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93817624,"statfs":{"total":96636764160,"available":96069246976,"internally_reserved":0,"allocated":258244608,"data_stored":258077423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,2,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.23799999999999999},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.55300000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.20200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":3.0409999999999999},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":8.8480000000000008}]}]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563848,"kb_used_data":261888,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93807992,"statfs":{"total":96636764160,"available":96059383808,"internally_reserved":0,"allocated":268173312,"data_stored":268006127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,4,2,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.42799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.216},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":6.0880000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.34399999999999997},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267235328,"data_stored":267235328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257966080,"data_stored":257966080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267046912,"data_stored":267046912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":267202560,"data_stored":267202560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:51.415 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:11:51.415 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7771", 'reported_seq': 7831, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.302567+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:11:48.302567+0000', 'last_peered': '2024-09-15T03:11:48.302567+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:11:48.302567+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7771, 'log_dups_size': 0, 'ondisk_log_size': 7771, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33906688, 'num_objects': 4139, 'num_object_clones': 0, 'num_object_copies': 16556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1968, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7771, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2171}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:51.415 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7559", 'reported_seq': 7649, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.260530+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:11:48.260530+0000', 'last_peered': '2024-09-15T03:11:48.260530+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:11:48.260530+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7559, 'log_dups_size': 0, 'ondisk_log_size': 7559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31676018, 'num_objects': 3867, 'num_object_clones': 0, 'num_object_copies': 15468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1987, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7557, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1882}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8050", 'reported_seq': 8154, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.600009+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:11:48.600009+0000', 'last_peered': '2024-09-15T03:11:48.600009+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:11:48.600009+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8050, 'log_dups_size': 0, 'ondisk_log_size': 8050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34159441, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 16684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2093, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8049, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2079}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7608", 'reported_seq': 11150, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.414523+0000', 'last_change': '2024-09-15T03:09:40.420402+0000', 'last_active': '2024-09-15T03:11:48.414523+0000', 'last_peered': '2024-09-15T03:11:48.414523+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:11:48.414523+0000', 'last_undegraded': '2024-09-15T03:06:08.878622+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7608, 'log_dups_size': 0, 'ondisk_log_size': 7608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32929394, 'num_objects': 4020, 'num_object_clones': 0, 'num_object_copies': 16080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 207, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7606, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1740, 'num_bytes_recovered': 14172160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7982", 'reported_seq': 8080, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.384929+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:11:48.384929+0000', 'last_peered': '2024-09-15T03:11:48.384929+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:11:48.384929+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7982, 'log_dups_size': 0, 'ondisk_log_size': 7982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33576562, 'num_objects': 4099, 'num_object_clones': 0, 'num_object_copies': 16396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2089, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7980, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2012}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7897", 'reported_seq': 7965, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.282852+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:11:48.282852+0000', 'last_peered': '2024-09-15T03:11:48.282852+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:11:48.282852+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7897, 'log_dups_size': 0, 'ondisk_log_size': 7897, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33595392, 'num_objects': 4101, 'num_object_clones': 0, 'num_object_copies': 16404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2045, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7897, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2056}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:51.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7871", 'reported_seq': 7980, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.401564+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:11:48.401564+0000', 'last_peered': '2024-09-15T03:11:48.401564+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:11:48.401564+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7871, 'log_dups_size': 0, 'ondisk_log_size': 7871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33478258, 'num_objects': 4087, 'num_object_clones': 0, 'num_object_copies': 16348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2033, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7869, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2056}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7959", 'reported_seq': 8074, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.417795+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:11:48.417795+0000', 'last_peered': '2024-09-15T03:11:48.417795+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:11:48.417795+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7959, 'log_dups_size': 0, 'ondisk_log_size': 7959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33797746, 'num_objects': 4126, 'num_object_clones': 0, 'num_object_copies': 16504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7957, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2070}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7856", 'reported_seq': 7941, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.376463+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:11:48.376463+0000', 'last_peered': '2024-09-15T03:11:48.376463+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:11:48.376463+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7856, 'log_dups_size': 0, 'ondisk_log_size': 7856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33696996, 'num_objects': 4114, 'num_object_clones': 0, 'num_object_copies': 16456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2032, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7853, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2085}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7883", 'reported_seq': 7996, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.501690+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:11:48.501690+0000', 'last_peered': '2024-09-15T03:11:48.501690+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:11:48.501690+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7883, 'log_dups_size': 0, 'ondisk_log_size': 7883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33461874, 'num_objects': 4085, 'num_object_clones': 0, 'num_object_copies': 16340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2046, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7881, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2041}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7844", 'reported_seq': 7926, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.413224+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:11:48.413224+0000', 'last_peered': '2024-09-15T03:11:48.413224+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:11:48.413224+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7844, 'log_dups_size': 0, 'ondisk_log_size': 7844, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33083819, 'num_objects': 4039, 'num_object_clones': 0, 'num_object_copies': 16156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2070, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7841, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1972}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:51.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:11:51.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7967", 'reported_seq': 8064, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.555745+0000', 'last_change': '2024-09-15T03:06:09.183465+0000', 'last_active': '2024-09-15T03:11:48.555745+0000', 'last_peered': '2024-09-15T03:11:48.555745+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:11:48.555745+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7967, 'log_dups_size': 0, 'ondisk_log_size': 7967, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34126649, 'num_objects': 4166, 'num_object_clones': 0, 'num_object_copies': 16664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7966, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2137}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:51.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:11:51.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7737", 'reported_seq': 7831, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.459033+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:11:48.459033+0000', 'last_peered': '2024-09-15T03:11:48.459033+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:11:48.459033+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7737, 'log_dups_size': 0, 'ondisk_log_size': 7737, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32314994, 'num_objects': 3945, 'num_object_clones': 0, 'num_object_copies': 15780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2017, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7735, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:11:51.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:11:51.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7780", 'reported_seq': 7877, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.597294+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:11:48.597294+0000', 'last_peered': '2024-09-15T03:11:48.597294+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:11:48.597294+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7780, 'log_dups_size': 0, 'ondisk_log_size': 7780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33609330, 'num_objects': 4103, 'num_object_clones': 0, 'num_object_copies': 16412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1983, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7778, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2122}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:11:51.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:11:51.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7925", 'reported_seq': 8030, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:48.272534+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:11:48.272534+0000', 'last_peered': '2024-09-15T03:11:48.272534+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:11:48.272534+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7925, 'log_dups_size': 0, 'ondisk_log_size': 7925, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33453682, 'num_objects': 4084, 'num_object_clones': 0, 'num_object_copies': 16336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2064, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7923, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2022}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:11:51.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:11:53.503 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:53.503 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28722/260984 objects degraded (11.005%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":436,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65246,"data_bytes":534452820,"bytes_used":2302582784,"bytes_avail":384244473856,"bytes_total":386547056640,"degraded_objects":28722,"degraded_total":260984,"degraded_ratio":0.11005272353860773,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":115301,"recovering_keys_per_sec":0,"num_objects_recovered":187,"num_bytes_recovered":1384448,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:11:53.560 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:11:53.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:54.964 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:55.006 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:55.019 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-15T03:11:54.816890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":525949524,"num_objects":64208,"num_object_clones":0,"num_object_copies":256832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64208,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126499,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4077,"num_bytes_recovered":30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126526,"ondisk_log_size":126526,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2240132,"kb_used_data":1024280,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375247228,"statfs":{"total":386547056640,"available":384253161472,"internally_reserved":0,"allocated":1048862720,"data_stored":1048197126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,6,8,12,6,9,0,0,1,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26271744,"num_objects":-3207,"num_object_clones":0,"num_object_copies":-12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3207,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":1802240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003286"},"pg_stats":[{"pgid":"1.f","version":"33'7845","reported_seq":7909,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.417690+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:53.417690+0000","last_peered":"2024-09-15T03:11:53.417690+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:53.417690+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7845,"log_dups_size":0,"ondisk_log_size":7845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33300480,"num_objects":4065,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7845,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2099}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7629","reported_seq":7721,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.254853+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:53.254853+0000","last_peered":"2024-09-15T03:11:53.254853+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:53.254853+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31102578,"num_objects":3797,"num_object_clones":0,"num_object_copies":15188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7627,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8118","reported_seq":8222,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.562333+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:53.562333+0000","last_peered":"2024-09-15T03:11:53.562333+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:53.562333+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8118,"log_dups_size":0,"ondisk_log_size":8118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33602385,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8117,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2011}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7674","reported_seq":11416,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.516419+0000","last_change":"2024-09-15T03:09:40.420402+0000","last_active":"2024-09-15T03:11:53.516419+0000","last_peered":"2024-09-15T03:11:53.516419+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:53.516419+0000","last_undegraded":"2024-09-15T03:06:08.878622+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7674,"log_dups_size":0,"ondisk_log_size":7674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32388722,"num_objects":3954,"num_object_clones":0,"num_object_copies":15816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7672,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3847}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8055","reported_seq":8155,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.470373+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:53.470373+0000","last_peered":"2024-09-15T03:11:53.470373+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:53.470373+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8055,"log_dups_size":0,"ondisk_log_size":8055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32978546,"num_objects":4026,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8053,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7960","reported_seq":8036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.460672+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:53.460672+0000","last_peered":"2024-09-15T03:11:53.460672+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:53.460672+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2041,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7960,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7937","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.394785+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:53.394785+0000","last_peered":"2024-09-15T03:11:53.394785+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:53.394785+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7937,"log_dups_size":0,"ondisk_log_size":7937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32937586,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7935,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1990}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8016","reported_seq":8131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.376628+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:53.376628+0000","last_peered":"2024-09-15T03:11:53.376628+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:53.376628+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33330802,"num_objects":4069,"num_object_clones":0,"num_object_copies":16276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7916","reported_seq":8005,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.136974+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:53.136974+0000","last_peered":"2024-09-15T03:11:53.136974+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:53.136974+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7916,"log_dups_size":0,"ondisk_log_size":7916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33205476,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7913,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2027}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7943","reported_seq":8056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.548110+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:53.548110+0000","last_peered":"2024-09-15T03:11:53.548110+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:53.548110+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32970354,"num_objects":4025,"num_object_clones":0,"num_object_copies":16100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1981}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7918","reported_seq":8002,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.492907+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:53.492907+0000","last_peered":"2024-09-15T03:11:53.492907+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:53.492907+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7918,"log_dups_size":0,"ondisk_log_size":7918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32477611,"num_objects":3965,"num_object_clones":0,"num_object_copies":15860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7915,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7851","reported_seq":11890,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:53.762438+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:53.762438+0000","last_peered":"2024-09-15T03:11:53.762438+0000","last_clean":"2024-09-15T03:11:53.762438+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:53.762438+0000","last_undegraded":"2024-09-15T03:11:53.762438+0000","last_fullsized":"2024-09-15T03:11:53.762438+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33159993,"num_objects":4048,"num_object_clones":0,"num_object_copies":16192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8032","reported_seq":8131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.723626+0000","last_change":"2024-09-15T03:06:09.183465+0000","last_active":"2024-09-15T03:11:53.723626+0000","last_peered":"2024-09-15T03:11:53.723626+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:53.723626+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8032,"log_dups_size":0,"ondisk_log_size":8032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33594169,"num_objects":4101,"num_object_clones":0,"num_object_copies":16404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2072}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7802","reported_seq":7898,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.471870+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:53.471870+0000","last_peered":"2024-09-15T03:11:53.471870+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:53.471870+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7802,"log_dups_size":0,"ondisk_log_size":7802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31782514,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1866}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7838","reported_seq":7935,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.557375+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:53.557375+0000","last_peered":"2024-09-15T03:11:53.557375+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:53.557375+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7838,"log_dups_size":0,"ondisk_log_size":7838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33134194,"num_objects":4045,"num_object_clones":0,"num_object_copies":16180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7836,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7992","reported_seq":8103,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:53.255561+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:53.255561+0000","last_peered":"2024-09-15T03:11:53.255561+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:53.255561+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7992,"log_dups_size":0,"ondisk_log_size":7992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32904818,"num_objects":4017,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7990,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":525949524,"num_objects":64208,"num_object_clones":0,"num_object_copies":256832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64208,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126499,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4077,"num_bytes_recovered":30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1046433792,"data_stored":1046433792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126526,"ondisk_log_size":126526,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555100,"kb_used_data":257236,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93816740,"statfs":{"total":96636764160,"available":96068341760,"internally_reserved":0,"allocated":263409664,"data_stored":263242479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,1,6,0,0,0,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.56599999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.28999999999999998},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.59899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558928,"kb_used_data":257408,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93812912,"statfs":{"total":96636764160,"available":96064421888,"internally_reserved":0,"allocated":263585792,"data_stored":263421753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,2,1,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.217}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.32700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":7.9390000000000001},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":7.907}]}]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562276,"kb_used_data":251932,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93809564,"statfs":{"total":96636764160,"available":96060993536,"internally_reserved":0,"allocated":257978368,"data_stored":257811183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":2.101},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":5.6840000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.29299999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563828,"kb_used_data":257704,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93808012,"statfs":{"total":96636764160,"available":96059404288,"internally_reserved":0,"allocated":263888896,"data_stored":263721711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":2.4399999999999999},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":3.8029999999999999},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":4.2199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.40699999999999997},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262975488,"data_stored":262975488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257683456,"data_stored":257683456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":262799360,"data_stored":262799360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262975488,"data_stored":262975488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:58.022 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:11:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:11:59.442 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:11:59.503 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-15T03:11:59.113990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517003860,"num_objects":63116,"num_object_clones":0,"num_object_copies":252464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127591,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4204,"num_bytes_recovered":30932992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127618,"ondisk_log_size":127618,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2234832,"kb_used_data":1010672,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375252528,"statfs":{"total":386547056640,"available":384258588672,"internally_reserved":0,"allocated":1034928128,"data_stored":1034262534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,3,12,10,4,4,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17448960,"num_objects":-2130,"num_object_clones":0,"num_object_copies":-8520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2130,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":442368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.298882"},"pg_stats":[{"pgid":"1.f","version":"33'7913","reported_seq":7977,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.333045+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:58.333045+0000","last_peered":"2024-09-15T03:11:58.333045+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:58.333045+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7913,"log_dups_size":0,"ondisk_log_size":7913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32743424,"num_objects":3997,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7913,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7690","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.507610+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:58.507610+0000","last_peered":"2024-09-15T03:11:58.507610+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:58.507610+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30602866,"num_objects":3736,"num_object_clones":0,"num_object_copies":14944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7688,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8192","reported_seq":8296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.548776+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:58.548776+0000","last_peered":"2024-09-15T03:11:58.548776+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:58.548776+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8192,"log_dups_size":0,"ondisk_log_size":8192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32996177,"num_objects":4029,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8191,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7738","reported_seq":11695,"reported_epoch":33,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:11:58.496164+0000","last_change":"2024-09-15T03:11:58.336609+0000","last_active":"2024-09-15T03:11:58.496164+0000","last_peered":"2024-09-15T03:11:58.496164+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:58.496164+0000","last_undegraded":"2024-09-15T03:11:58.496164+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864434,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","1(0)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8125","reported_seq":8229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.445126+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:58.445126+0000","last_peered":"2024-09-15T03:11:58.445126+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:58.445126+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8125,"log_dups_size":0,"ondisk_log_size":8125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32405106,"num_objects":3956,"num_object_clones":0,"num_object_copies":15824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8123,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8027","reported_seq":8107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.537723+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:58.537723+0000","last_peered":"2024-09-15T03:11:58.537723+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:58.537723+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8027,"log_dups_size":0,"ondisk_log_size":8027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32530432,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7999","reported_seq":8112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.167902+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:58.167902+0000","last_peered":"2024-09-15T03:11:58.167902+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:58.167902+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32429682,"num_objects":3959,"num_object_clones":0,"num_object_copies":15836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7997,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1930}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8093","reported_seq":8207,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.509144+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:58.509144+0000","last_peered":"2024-09-15T03:11:58.509144+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:58.509144+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8093,"log_dups_size":0,"ondisk_log_size":8093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32700018,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8091,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7995","reported_seq":8084,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.415780+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:58.415780+0000","last_peered":"2024-09-15T03:11:58.415780+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:58.415780+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32558308,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8022","reported_seq":8139,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.725030+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:58.725030+0000","last_peered":"2024-09-15T03:11:58.725030+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:58.725030+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8022,"log_dups_size":0,"ondisk_log_size":8022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32323186,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1904}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7996","reported_seq":8086,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.491912+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:58.491912+0000","last_peered":"2024-09-15T03:11:58.491912+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:58.491912+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7996,"log_dups_size":0,"ondisk_log_size":7996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31838635,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7920","reported_seq":11959,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:58.831629+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:58.831629+0000","last_peered":"2024-09-15T03:11:58.831629+0000","last_clean":"2024-09-15T03:11:58.831629+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:58.831629+0000","last_undegraded":"2024-09-15T03:11:58.831629+0000","last_fullsized":"2024-09-15T03:11:58.831629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7920,"log_dups_size":0,"ondisk_log_size":7920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32594745,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7919,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8087","reported_seq":8195,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.674623+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:11:58.674623+0000","last_peered":"2024-09-15T03:11:58.674623+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:58.674623+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33143609,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7856","reported_seq":7954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.119110+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:58.119110+0000","last_peered":"2024-09-15T03:11:58.119110+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:58.119110+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31340146,"num_objects":3826,"num_object_clones":0,"num_object_copies":15304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7854,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7904","reported_seq":8009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.731103+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:58.731103+0000","last_peered":"2024-09-15T03:11:58.731103+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:58.731103+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7904,"log_dups_size":0,"ondisk_log_size":7904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32593522,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7902,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2002}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8061","reported_seq":8172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.446531+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:58.446531+0000","last_peered":"2024-09-15T03:11:58.446531+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:58.446531+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32339570,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517003860,"num_objects":63116,"num_object_clones":0,"num_object_copies":252464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127591,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4204,"num_bytes_recovered":30932992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1032429568,"data_stored":1032429568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127618,"ondisk_log_size":127618,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559068,"kb_used_data":252884,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93812772,"statfs":{"total":96636764160,"available":96064278528,"internally_reserved":0,"allocated":258953216,"data_stored":258786031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,5,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.353}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":9.9920000000000009},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.59199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562840,"kb_used_data":253008,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93809000,"statfs":{"total":96636764160,"available":96060416000,"internally_reserved":0,"allocated":259080192,"data_stored":258916153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,5,1,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.39800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":14.709},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":14.880000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549276,"kb_used_data":251412,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93822564,"statfs":{"total":96636764160,"available":96074305536,"internally_reserved":0,"allocated":257445888,"data_stored":257278703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.78100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.56999999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":2.7200000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563648,"kb_used_data":253368,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93808192,"statfs":{"total":96636764160,"available":96059588608,"internally_reserved":0,"allocated":259448832,"data_stored":259281647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":7.9470000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":7.6689999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.247},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.46000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.84299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258473984,"data_stored":258473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257150976,"data_stored":257150976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258330624,"data_stored":258330624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258473984,"data_stored":258473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:11:59.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:00.846 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:00.870 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:00.894 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-15T03:12:00.818471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517003860,"num_objects":63116,"num_object_clones":0,"num_object_copies":252464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127591,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4204,"num_bytes_recovered":30932992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127618,"ondisk_log_size":127618,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2234832,"kb_used_data":1010672,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375252528,"statfs":{"total":386547056640,"available":384258588672,"internally_reserved":0,"allocated":1034928128,"data_stored":1034262534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,3,12,10,4,4,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17448960,"num_objects":-2130,"num_object_clones":0,"num_object_copies":-8520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2130,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":442368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002065"},"pg_stats":[{"pgid":"1.f","version":"33'7913","reported_seq":7977,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.333045+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:58.333045+0000","last_peered":"2024-09-15T03:11:58.333045+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:58.333045+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7913,"log_dups_size":0,"ondisk_log_size":7913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32743424,"num_objects":3997,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7913,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7690","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.507610+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:58.507610+0000","last_peered":"2024-09-15T03:11:58.507610+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:58.507610+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30602866,"num_objects":3736,"num_object_clones":0,"num_object_copies":14944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7688,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8192","reported_seq":8296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.548776+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:58.548776+0000","last_peered":"2024-09-15T03:11:58.548776+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:58.548776+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8192,"log_dups_size":0,"ondisk_log_size":8192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32996177,"num_objects":4029,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8191,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7738","reported_seq":11695,"reported_epoch":33,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:11:58.496164+0000","last_change":"2024-09-15T03:11:58.336609+0000","last_active":"2024-09-15T03:11:58.496164+0000","last_peered":"2024-09-15T03:11:58.496164+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:58.496164+0000","last_undegraded":"2024-09-15T03:11:58.496164+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864434,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","1(0)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8125","reported_seq":8229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.445126+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:58.445126+0000","last_peered":"2024-09-15T03:11:58.445126+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:58.445126+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8125,"log_dups_size":0,"ondisk_log_size":8125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32405106,"num_objects":3956,"num_object_clones":0,"num_object_copies":15824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8123,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8027","reported_seq":8107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.537723+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:58.537723+0000","last_peered":"2024-09-15T03:11:58.537723+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:58.537723+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8027,"log_dups_size":0,"ondisk_log_size":8027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32530432,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7999","reported_seq":8112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.167902+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:58.167902+0000","last_peered":"2024-09-15T03:11:58.167902+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:58.167902+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32429682,"num_objects":3959,"num_object_clones":0,"num_object_copies":15836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7997,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1930}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8093","reported_seq":8207,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.509144+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:58.509144+0000","last_peered":"2024-09-15T03:11:58.509144+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:58.509144+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8093,"log_dups_size":0,"ondisk_log_size":8093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32700018,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8091,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7995","reported_seq":8084,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.415780+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:58.415780+0000","last_peered":"2024-09-15T03:11:58.415780+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:58.415780+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32558308,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8022","reported_seq":8139,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.725030+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:58.725030+0000","last_peered":"2024-09-15T03:11:58.725030+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:58.725030+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8022,"log_dups_size":0,"ondisk_log_size":8022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32323186,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1904}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7996","reported_seq":8086,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.491912+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:58.491912+0000","last_peered":"2024-09-15T03:11:58.491912+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:58.491912+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7996,"log_dups_size":0,"ondisk_log_size":7996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31838635,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7920","reported_seq":11959,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:58.831629+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:58.831629+0000","last_peered":"2024-09-15T03:11:58.831629+0000","last_clean":"2024-09-15T03:11:58.831629+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:58.831629+0000","last_undegraded":"2024-09-15T03:11:58.831629+0000","last_fullsized":"2024-09-15T03:11:58.831629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7920,"log_dups_size":0,"ondisk_log_size":7920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32594745,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7919,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8087","reported_seq":8195,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.674623+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:11:58.674623+0000","last_peered":"2024-09-15T03:11:58.674623+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:58.674623+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33143609,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7856","reported_seq":7954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.119110+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:58.119110+0000","last_peered":"2024-09-15T03:11:58.119110+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:58.119110+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31340146,"num_objects":3826,"num_object_clones":0,"num_object_copies":15304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7854,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7904","reported_seq":8009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.731103+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:58.731103+0000","last_peered":"2024-09-15T03:11:58.731103+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:58.731103+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7904,"log_dups_size":0,"ondisk_log_size":7904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32593522,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7902,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2002}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8061","reported_seq":8172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.446531+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:58.446531+0000","last_peered":"2024-09-15T03:11:58.446531+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:58.446531+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32339570,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517003860,"num_objects":63116,"num_object_clones":0,"num_object_copies":252464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127591,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4204,"num_bytes_recovered":30932992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1032429568,"data_stored":1032429568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127618,"ondisk_log_size":127618,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559068,"kb_used_data":252884,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93812772,"statfs":{"total":96636764160,"available":96064278528,"internally_reserved":0,"allocated":258953216,"data_stored":258786031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,5,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.353}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":9.9920000000000009},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.59199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562840,"kb_used_data":253008,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93809000,"statfs":{"total":96636764160,"available":96060416000,"internally_reserved":0,"allocated":259080192,"data_stored":258916153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,5,1,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.39800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":14.709},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":14.880000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549276,"kb_used_data":251412,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93822564,"statfs":{"total":96636764160,"available":96074305536,"internally_reserved":0,"allocated":257445888,"data_stored":257278703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.78100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.56999999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":2.7200000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563648,"kb_used_data":253368,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93808192,"statfs":{"total":96636764160,"available":96059588608,"internally_reserved":0,"allocated":259448832,"data_stored":259281647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":7.9470000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":7.6689999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.247},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.46000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.84299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258473984,"data_stored":258473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257150976,"data_stored":257150976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258330624,"data_stored":258330624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258473984,"data_stored":258473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:00.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7913", 'reported_seq': 7977, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.333045+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:11:58.333045+0000', 'last_peered': '2024-09-15T03:11:58.333045+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:11:58.333045+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7913, 'log_dups_size': 0, 'ondisk_log_size': 7913, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32743424, 'num_objects': 3997, 'num_object_clones': 0, 'num_object_copies': 15988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7913, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7690", 'reported_seq': 7782, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.507610+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:11:58.507610+0000', 'last_peered': '2024-09-15T03:11:58.507610+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:11:58.507610+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7690, 'log_dups_size': 0, 'ondisk_log_size': 7690, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30602866, 'num_objects': 3736, 'num_object_clones': 0, 'num_object_copies': 14944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7688, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8192", 'reported_seq': 8296, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.548776+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:11:58.548776+0000', 'last_peered': '2024-09-15T03:11:58.548776+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:11:58.548776+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8192, 'log_dups_size': 0, 'ondisk_log_size': 8192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32996177, 'num_objects': 4029, 'num_object_clones': 0, 'num_object_copies': 16116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2093, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4029, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8191, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1937}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7738", 'reported_seq': 11695, 'reported_epoch': 33, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-15T03:11:58.496164+0000', 'last_change': '2024-09-15T03:11:58.336609+0000', 'last_active': '2024-09-15T03:11:58.496164+0000', 'last_peered': '2024-09-15T03:11:58.496164+0000', 'last_clean': '2024-09-15T03:05:04.906336+0000', 'last_became_active': '2024-09-15T03:06:09.192258+0000', 'last_became_peered': '2024-09-15T03:06:09.192258+0000', 'last_unstale': '2024-09-15T03:11:58.496164+0000', 'last_undegraded': '2024-09-15T03:11:58.496164+0000', 'last_fullsized': '2024-09-15T03:06:08.840393+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7738, 'log_dups_size': 0, 'ondisk_log_size': 7738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31864434, 'num_objects': 3890, 'num_object_clones': 0, 'num_object_copies': 15560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7736, 'num_write_kb': 46501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1947, 'num_bytes_recovered': 14589952, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '1(0)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:12:00.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8125", 'reported_seq': 8229, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.445126+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:11:58.445126+0000', 'last_peered': '2024-09-15T03:11:58.445126+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:11:58.445126+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8125, 'log_dups_size': 0, 'ondisk_log_size': 8125, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32405106, 'num_objects': 3956, 'num_object_clones': 0, 'num_object_copies': 15824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8123, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1872}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:00.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:12:00.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8027", 'reported_seq': 8107, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.537723+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:11:58.537723+0000', 'last_peered': '2024-09-15T03:11:58.537723+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:11:58.537723+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8027, 'log_dups_size': 0, 'ondisk_log_size': 8027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32530432, 'num_objects': 3971, 'num_object_clones': 0, 'num_object_copies': 15884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2039, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8027, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1932}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:00.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:12:00.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7999", 'reported_seq': 8112, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.167902+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:11:58.167902+0000', 'last_peered': '2024-09-15T03:11:58.167902+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:11:58.167902+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7999, 'log_dups_size': 0, 'ondisk_log_size': 7999, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32429682, 'num_objects': 3959, 'num_object_clones': 0, 'num_object_copies': 15836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2031, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3959, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7997, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1930}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:00.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:12:00.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8093", 'reported_seq': 8207, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.509144+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:11:58.509144+0000', 'last_peered': '2024-09-15T03:11:58.509144+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:11:58.509144+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8093, 'log_dups_size': 0, 'ondisk_log_size': 8093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32700018, 'num_objects': 3992, 'num_object_clones': 0, 'num_object_copies': 15968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8091, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1936}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7995", 'reported_seq': 8084, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.415780+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:11:58.415780+0000', 'last_peered': '2024-09-15T03:11:58.415780+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:11:58.415780+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7995, 'log_dups_size': 0, 'ondisk_log_size': 7995, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32558308, 'num_objects': 3975, 'num_object_clones': 0, 'num_object_copies': 15900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7992, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1948}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8022", 'reported_seq': 8139, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.725030+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:11:58.725030+0000', 'last_peered': '2024-09-15T03:11:58.725030+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:11:58.725030+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8022, 'log_dups_size': 0, 'ondisk_log_size': 8022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32323186, 'num_objects': 3946, 'num_object_clones': 0, 'num_object_copies': 15784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8020, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1904}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'7996", 'reported_seq': 8086, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.491912+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:11:58.491912+0000', 'last_peered': '2024-09-15T03:11:58.491912+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:11:58.491912+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7996, 'log_dups_size': 0, 'ondisk_log_size': 7996, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31838635, 'num_objects': 3887, 'num_object_clones': 0, 'num_object_copies': 15548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2066, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7993, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1824}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8087", 'reported_seq': 8195, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.674623+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:11:58.674623+0000', 'last_peered': '2024-09-15T03:11:58.674623+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:11:58.674623+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8087, 'log_dups_size': 0, 'ondisk_log_size': 8087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33143609, 'num_objects': 4046, 'num_object_clones': 0, 'num_object_copies': 16184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2026, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8086, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 24576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2020}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:12:00.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7856", 'reported_seq': 7954, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.119110+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:11:58.119110+0000', 'last_peered': '2024-09-15T03:11:58.119110+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:11:58.119110+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7856, 'log_dups_size': 0, 'ondisk_log_size': 7856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31340146, 'num_objects': 3826, 'num_object_clones': 0, 'num_object_copies': 15304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7854, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:00.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:12:00.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7904", 'reported_seq': 8009, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.731103+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:11:58.731103+0000', 'last_peered': '2024-09-15T03:11:58.731103+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:11:58.731103+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7904, 'log_dups_size': 0, 'ondisk_log_size': 7904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32593522, 'num_objects': 3979, 'num_object_clones': 0, 'num_object_copies': 15916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7902, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2002}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:00.899 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:12:00.899 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8061", 'reported_seq': 8172, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:11:58.446531+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:11:58.446531+0000', 'last_peered': '2024-09-15T03:11:58.446531+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:11:58.446531+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8061, 'log_dups_size': 0, 'ondisk_log_size': 8061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32339570, 'num_objects': 3948, 'num_object_clones': 0, 'num_object_copies': 15792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8059, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1889}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:00.899 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:12:02.754 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:02.754 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28480/252464 objects degraded (11.281%), 14 pgs degraded, 15 pgs undersized","count":29},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":445,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+undersized+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63116,"data_bytes":517003860,"bytes_used":2288467968,"bytes_avail":384258588672,"bytes_total":386547056640,"degraded_objects":28480,"degraded_total":252464,"degraded_ratio":0.11280816274795616,"recovering_objects_per_sec":30,"recovering_bytes_per_sec":55281,"recovering_keys_per_sec":0,"num_objects_recovered":242,"num_bytes_recovered":442368,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":266},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:12:02.834 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:12:02.834 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:04.634 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:04.640 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:04.655 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-15T03:12:02.822866+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":517003860,"num_objects":63116,"num_object_clones":0,"num_object_copies":252464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127591,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4204,"num_bytes_recovered":30932992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127618,"ondisk_log_size":127618,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2234832,"kb_used_data":1010672,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375252528,"statfs":{"total":386547056640,"available":384258588672,"internally_reserved":0,"allocated":1034928128,"data_stored":1034262534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,3,12,10,4,4,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8945664,"num_objects":-1092,"num_object_clones":0,"num_object_copies":-4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1092,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005976"},"pg_stats":[{"pgid":"1.f","version":"33'7913","reported_seq":7977,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.333045+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:11:58.333045+0000","last_peered":"2024-09-15T03:11:58.333045+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:11:58.333045+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7913,"log_dups_size":0,"ondisk_log_size":7913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32743424,"num_objects":3997,"num_object_clones":0,"num_object_copies":15988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7913,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7690","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.507610+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:11:58.507610+0000","last_peered":"2024-09-15T03:11:58.507610+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:11:58.507610+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7690,"log_dups_size":0,"ondisk_log_size":7690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30602866,"num_objects":3736,"num_object_clones":0,"num_object_copies":14944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7688,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1752}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8192","reported_seq":8296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.548776+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:11:58.548776+0000","last_peered":"2024-09-15T03:11:58.548776+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:11:58.548776+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8192,"log_dups_size":0,"ondisk_log_size":8192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32996177,"num_objects":4029,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8191,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7738","reported_seq":11695,"reported_epoch":33,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:11:58.496164+0000","last_change":"2024-09-15T03:11:58.336609+0000","last_active":"2024-09-15T03:11:58.496164+0000","last_peered":"2024-09-15T03:11:58.496164+0000","last_clean":"2024-09-15T03:05:04.906336+0000","last_became_active":"2024-09-15T03:06:09.192258+0000","last_became_peered":"2024-09-15T03:06:09.192258+0000","last_unstale":"2024-09-15T03:11:58.496164+0000","last_undegraded":"2024-09-15T03:11:58.496164+0000","last_fullsized":"2024-09-15T03:06:08.840393+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864434,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","1(0)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8125","reported_seq":8229,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.445126+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:11:58.445126+0000","last_peered":"2024-09-15T03:11:58.445126+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:11:58.445126+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8125,"log_dups_size":0,"ondisk_log_size":8125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32405106,"num_objects":3956,"num_object_clones":0,"num_object_copies":15824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8123,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8027","reported_seq":8107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.537723+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:11:58.537723+0000","last_peered":"2024-09-15T03:11:58.537723+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:11:58.537723+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8027,"log_dups_size":0,"ondisk_log_size":8027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32530432,"num_objects":3971,"num_object_clones":0,"num_object_copies":15884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7999","reported_seq":8112,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.167902+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:11:58.167902+0000","last_peered":"2024-09-15T03:11:58.167902+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:11:58.167902+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32429682,"num_objects":3959,"num_object_clones":0,"num_object_copies":15836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7997,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1930}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8093","reported_seq":8207,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.509144+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:11:58.509144+0000","last_peered":"2024-09-15T03:11:58.509144+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:11:58.509144+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8093,"log_dups_size":0,"ondisk_log_size":8093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32700018,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8091,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7995","reported_seq":8084,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.415780+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:11:58.415780+0000","last_peered":"2024-09-15T03:11:58.415780+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:11:58.415780+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7995,"log_dups_size":0,"ondisk_log_size":7995,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32558308,"num_objects":3975,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1948}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8022","reported_seq":8139,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.725030+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:11:58.725030+0000","last_peered":"2024-09-15T03:11:58.725030+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:11:58.725030+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8022,"log_dups_size":0,"ondisk_log_size":8022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32323186,"num_objects":3946,"num_object_clones":0,"num_object_copies":15784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1904}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7996","reported_seq":8086,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.491912+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:11:58.491912+0000","last_peered":"2024-09-15T03:11:58.491912+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:11:58.491912+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7996,"log_dups_size":0,"ondisk_log_size":7996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31838635,"num_objects":3887,"num_object_clones":0,"num_object_copies":15548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7993,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7920","reported_seq":11959,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T03:11:58.831629+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:11:58.831629+0000","last_peered":"2024-09-15T03:11:58.831629+0000","last_clean":"2024-09-15T03:11:58.831629+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:11:58.831629+0000","last_undegraded":"2024-09-15T03:11:58.831629+0000","last_fullsized":"2024-09-15T03:11:58.831629+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7920,"log_dups_size":0,"ondisk_log_size":7920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32594745,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7919,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"33'8087","reported_seq":8195,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.674623+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:11:58.674623+0000","last_peered":"2024-09-15T03:11:58.674623+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:11:58.674623+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33143609,"num_objects":4046,"num_object_clones":0,"num_object_copies":16184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8086,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":24576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7856","reported_seq":7954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.119110+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:11:58.119110+0000","last_peered":"2024-09-15T03:11:58.119110+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:11:58.119110+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31340146,"num_objects":3826,"num_object_clones":0,"num_object_copies":15304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7854,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1813}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7904","reported_seq":8009,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.731103+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:11:58.731103+0000","last_peered":"2024-09-15T03:11:58.731103+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:11:58.731103+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7904,"log_dups_size":0,"ondisk_log_size":7904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32593522,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7902,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2002}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8061","reported_seq":8172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:11:58.446531+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:11:58.446531+0000","last_peered":"2024-09-15T03:11:58.446531+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:11:58.446531+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32339570,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":517003860,"num_objects":63116,"num_object_clones":0,"num_object_copies":252464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127591,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4204,"num_bytes_recovered":30932992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1032429568,"data_stored":1032429568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127618,"ondisk_log_size":127618,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559068,"kb_used_data":252884,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93812772,"statfs":{"total":96636764160,"available":96064278528,"internally_reserved":0,"allocated":258953216,"data_stored":258786031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,5,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.353}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":9.9920000000000009},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.59199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562840,"kb_used_data":253008,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93809000,"statfs":{"total":96636764160,"available":96060416000,"internally_reserved":0,"allocated":259080192,"data_stored":258916153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,5,1,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.48799999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.39800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":14.709},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":14.880000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549276,"kb_used_data":251412,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93822564,"statfs":{"total":96636764160,"available":96074305536,"internally_reserved":0,"allocated":257445888,"data_stored":257278703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":0.78100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":0.56999999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":2.7200000000000002},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.46100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563648,"kb_used_data":253368,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93808192,"statfs":{"total":96636764160,"available":96059588608,"internally_reserved":0,"allocated":259448832,"data_stored":259281647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":7.9470000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":7.6689999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.247},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.46000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.84299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258473984,"data_stored":258473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":257150976,"data_stored":257150976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":258330624,"data_stored":258330624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258473984,"data_stored":258473984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:07.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:09.185 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:09.244 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:09.271 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-15T03:12:08.828406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503503444,"num_objects":61468,"num_object_clones":0,"num_object_copies":245872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129239,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4269,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129266,"ondisk_log_size":129266,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2194692,"kb_used_data":987164,"kb_used_omap":14,"kb_used_meta":1207473,"kb_avail":375292668,"statfs":{"total":386547056640,"available":384299692032,"internally_reserved":0,"allocated":1010855936,"data_stored":1010157218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236452379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,9,14,11,13,5,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13500416,"num_objects":-1648,"num_object_clones":0,"num_object_copies":-6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":139264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.714416"},"pg_stats":[{"pgid":"1.f","version":"35'8046","reported_seq":8112,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.666881+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:08.666881+0000","last_peered":"2024-09-15T03:12:08.666881+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:08.666881+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31653888,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7810","reported_seq":7904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.555614+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:08.555614+0000","last_peered":"2024-09-15T03:12:08.555614+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:08.555614+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7810,"log_dups_size":0,"ondisk_log_size":7810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29619826,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8257","reported_seq":8363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:03.721328+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:03.721328+0000","last_peered":"2024-09-15T03:12:03.721328+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:03.721328+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8257,"log_dups_size":0,"ondisk_log_size":8257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32463697,"num_objects":3964,"num_object_clones":0,"num_object_copies":15856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8256,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1872}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7802","reported_seq":11774,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:03.897709+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:03.897709+0000","last_peered":"2024-09-15T03:12:03.897709+0000","last_clean":"2024-09-15T03:12:03.897709+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:03.897709+0000","last_undegraded":"2024-09-15T03:12:03.897709+0000","last_fullsized":"2024-09-15T03:12:03.897709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7802,"log_dups_size":0,"ondisk_log_size":7802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31340146,"num_objects":3826,"num_object_clones":0,"num_object_copies":15304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8246","reported_seq":8352,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.603056+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:08.603056+0000","last_peered":"2024-09-15T03:12:08.603056+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:08.603056+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31413874,"num_objects":3835,"num_object_clones":0,"num_object_copies":15340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8152","reported_seq":8236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.368854+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:08.368854+0000","last_peered":"2024-09-15T03:12:08.368854+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:08.368854+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8152,"log_dups_size":0,"ondisk_log_size":8152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31506432,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8126","reported_seq":8247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.484234+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:08.484234+0000","last_peered":"2024-09-15T03:12:08.484234+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:08.484234+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31389298,"num_objects":3832,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8226","reported_seq":8343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.496189+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:08.496189+0000","last_peered":"2024-09-15T03:12:08.496189+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:08.496189+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31610482,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8127","reported_seq":8218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.622774+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:08.622774+0000","last_peered":"2024-09-15T03:12:08.622774+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:08.622774+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31476964,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8092","reported_seq":8206,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:03.650727+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:03.650727+0000","last_peered":"2024-09-15T03:12:03.650727+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:03.650727+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31749746,"num_objects":3876,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8136","reported_seq":8230,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.463835+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:08.463835+0000","last_peered":"2024-09-15T03:12:08.463835+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:08.463835+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30691755,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7985","reported_seq":12026,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:03.847184+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:03.847184+0000","last_peered":"2024-09-15T03:12:03.847184+0000","last_clean":"2024-09-15T03:12:03.847184+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:03.847184+0000","last_undegraded":"2024-09-15T03:12:03.847184+0000","last_fullsized":"2024-09-15T03:12:03.847184+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32062265,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7984,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8141","reported_seq":8372,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:03.773074+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:03.773074+0000","last_peered":"2024-09-15T03:12:03.773074+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:03.773074+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32701241,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8140,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2026}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7975","reported_seq":8075,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.485550+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:08.485550+0000","last_peered":"2024-09-15T03:12:08.485550+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:08.485550+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30365298,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7978","reported_seq":8085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:03.618417+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:03.618417+0000","last_peered":"2024-09-15T03:12:03.618417+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:03.618417+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7978,"log_dups_size":0,"ondisk_log_size":7978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31987314,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7976,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8167","reported_seq":8278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.516723+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:08.516723+0000","last_peered":"2024-09-15T03:12:08.516723+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:08.516723+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8167,"log_dups_size":0,"ondisk_log_size":8167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31471218,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":503503444,"num_objects":61468,"num_object_clones":0,"num_object_copies":245872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129239,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4269,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1008492544,"data_stored":1008492544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129266,"ondisk_log_size":129266,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542492,"kb_used_data":244624,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93829348,"statfs":{"total":96636764160,"available":96081252352,"internally_reserved":0,"allocated":250494976,"data_stored":250319510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,2,2,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.249}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.38500000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546440,"kb_used_data":249088,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93825400,"statfs":{"total":96636764160,"available":96077209600,"internally_reserved":0,"allocated":255066112,"data_stored":254893792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,6,1,1,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.40699999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.45400000000000001},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.48399999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":2.3170000000000002},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":2.2909999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555128,"kb_used_data":248944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93816712,"statfs":{"total":96636764160,"available":96068313088,"internally_reserved":0,"allocated":254918656,"data_stored":254743190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":10.343},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":5.1020000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":3.79},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":16.690000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.621},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":3.8279999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550632,"kb_used_data":244508,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93821208,"statfs":{"total":96636764160,"available":96072916992,"internally_reserved":0,"allocated":250376192,"data_stored":250200726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,0,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.50900000000000001},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.27900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":30.013000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249999360,"data_stored":249999360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254255104,"data_stored":254255104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254218240,"data_stored":254218240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250019840,"data_stored":250019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:09.273 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:10.503 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:10.517 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:10.538 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-15T03:12:08.828406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503503444,"num_objects":61468,"num_object_clones":0,"num_object_copies":245872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129239,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4269,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129266,"ondisk_log_size":129266,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2194692,"kb_used_data":987164,"kb_used_omap":14,"kb_used_meta":1207473,"kb_avail":375292668,"statfs":{"total":386547056640,"available":384299692032,"internally_reserved":0,"allocated":1010855936,"data_stored":1010157218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236452379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,9,14,11,13,5,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13500416,"num_objects":-1648,"num_object_clones":0,"num_object_copies":-6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":139264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.714416"},"pg_stats":[{"pgid":"1.f","version":"35'8046","reported_seq":8112,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.666881+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:08.666881+0000","last_peered":"2024-09-15T03:12:08.666881+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:08.666881+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31653888,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7810","reported_seq":7904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.555614+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:08.555614+0000","last_peered":"2024-09-15T03:12:08.555614+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:08.555614+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7810,"log_dups_size":0,"ondisk_log_size":7810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29619826,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8257","reported_seq":8363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:03.721328+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:03.721328+0000","last_peered":"2024-09-15T03:12:03.721328+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:03.721328+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8257,"log_dups_size":0,"ondisk_log_size":8257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32463697,"num_objects":3964,"num_object_clones":0,"num_object_copies":15856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8256,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1872}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7802","reported_seq":11774,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:03.897709+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:03.897709+0000","last_peered":"2024-09-15T03:12:03.897709+0000","last_clean":"2024-09-15T03:12:03.897709+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:03.897709+0000","last_undegraded":"2024-09-15T03:12:03.897709+0000","last_fullsized":"2024-09-15T03:12:03.897709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7802,"log_dups_size":0,"ondisk_log_size":7802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31340146,"num_objects":3826,"num_object_clones":0,"num_object_copies":15304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8246","reported_seq":8352,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.603056+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:08.603056+0000","last_peered":"2024-09-15T03:12:08.603056+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:08.603056+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31413874,"num_objects":3835,"num_object_clones":0,"num_object_copies":15340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8152","reported_seq":8236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.368854+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:08.368854+0000","last_peered":"2024-09-15T03:12:08.368854+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:08.368854+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8152,"log_dups_size":0,"ondisk_log_size":8152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31506432,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8126","reported_seq":8247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.484234+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:08.484234+0000","last_peered":"2024-09-15T03:12:08.484234+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:08.484234+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31389298,"num_objects":3832,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8226","reported_seq":8343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.496189+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:08.496189+0000","last_peered":"2024-09-15T03:12:08.496189+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:08.496189+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31610482,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8127","reported_seq":8218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.622774+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:08.622774+0000","last_peered":"2024-09-15T03:12:08.622774+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:08.622774+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31476964,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8092","reported_seq":8206,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:03.650727+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:03.650727+0000","last_peered":"2024-09-15T03:12:03.650727+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:03.650727+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8092,"log_dups_size":0,"ondisk_log_size":8092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31749746,"num_objects":3876,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1834}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8136","reported_seq":8230,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.463835+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:08.463835+0000","last_peered":"2024-09-15T03:12:08.463835+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:08.463835+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30691755,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'7985","reported_seq":12026,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:03.847184+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:03.847184+0000","last_peered":"2024-09-15T03:12:03.847184+0000","last_clean":"2024-09-15T03:12:03.847184+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:03.847184+0000","last_undegraded":"2024-09-15T03:12:03.847184+0000","last_fullsized":"2024-09-15T03:12:03.847184+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7985,"log_dups_size":0,"ondisk_log_size":7985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32062265,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7984,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8141","reported_seq":8372,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:03.773074+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:03.773074+0000","last_peered":"2024-09-15T03:12:03.773074+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:03.773074+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32701241,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8140,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2026}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7975","reported_seq":8075,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.485550+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:08.485550+0000","last_peered":"2024-09-15T03:12:08.485550+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:08.485550+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30365298,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'7978","reported_seq":8085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:03.618417+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:03.618417+0000","last_peered":"2024-09-15T03:12:03.618417+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:03.618417+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7978,"log_dups_size":0,"ondisk_log_size":7978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31987314,"num_objects":3905,"num_object_clones":0,"num_object_copies":15620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7976,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8167","reported_seq":8278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.516723+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:08.516723+0000","last_peered":"2024-09-15T03:12:08.516723+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:08.516723+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8167,"log_dups_size":0,"ondisk_log_size":8167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31471218,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":503503444,"num_objects":61468,"num_object_clones":0,"num_object_copies":245872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61468,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129239,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4269,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1008492544,"data_stored":1008492544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129266,"ondisk_log_size":129266,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542492,"kb_used_data":244624,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93829348,"statfs":{"total":96636764160,"available":96081252352,"internally_reserved":0,"allocated":250494976,"data_stored":250319510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,2,2,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.249}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.38500000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546440,"kb_used_data":249088,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93825400,"statfs":{"total":96636764160,"available":96077209600,"internally_reserved":0,"allocated":255066112,"data_stored":254893792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,6,1,1,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.40699999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.45400000000000001},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.48399999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":2.3170000000000002},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":2.2909999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555128,"kb_used_data":248944,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93816712,"statfs":{"total":96636764160,"available":96068313088,"internally_reserved":0,"allocated":254918656,"data_stored":254743190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":10.343},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":5.1020000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":3.79},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":16.690000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":0.621},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":3.8279999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550632,"kb_used_data":244508,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93821208,"statfs":{"total":96636764160,"available":96072916992,"internally_reserved":0,"allocated":250376192,"data_stored":250200726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,0,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.50900000000000001},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.27900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":30.013000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249999360,"data_stored":249999360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254255104,"data_stored":254255104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":254218240,"data_stored":254218240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250019840,"data_stored":250019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:10.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:12:10.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8046", 'reported_seq': 8112, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.666881+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:12:08.666881+0000', 'last_peered': '2024-09-15T03:12:08.666881+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:12:08.666881+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8046, 'log_dups_size': 0, 'ondisk_log_size': 8046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31653888, 'num_objects': 3864, 'num_object_clones': 0, 'num_object_copies': 15456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8046, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1898}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:10.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:12:10.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7810", 'reported_seq': 7904, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.555614+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:12:08.555614+0000', 'last_peered': '2024-09-15T03:12:08.555614+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:12:08.555614+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7810, 'log_dups_size': 0, 'ondisk_log_size': 7810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29619826, 'num_objects': 3616, 'num_object_clones': 0, 'num_object_copies': 14464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7808, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8257", 'reported_seq': 8363, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:03.721328+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:12:03.721328+0000', 'last_peered': '2024-09-15T03:12:03.721328+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:12:03.721328+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8257, 'log_dups_size': 0, 'ondisk_log_size': 8257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32463697, 'num_objects': 3964, 'num_object_clones': 0, 'num_object_copies': 15856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2093, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3964, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8256, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1872}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8246", 'reported_seq': 8352, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.603056+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:12:08.603056+0000', 'last_peered': '2024-09-15T03:12:08.603056+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:12:08.603056+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8246, 'log_dups_size': 0, 'ondisk_log_size': 8246, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31413874, 'num_objects': 3835, 'num_object_clones': 0, 'num_object_copies': 15340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8244, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1751}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8152", 'reported_seq': 8236, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.368854+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:12:08.368854+0000', 'last_peered': '2024-09-15T03:12:08.368854+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:12:08.368854+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8152, 'log_dups_size': 0, 'ondisk_log_size': 8152, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31506432, 'num_objects': 3846, 'num_object_clones': 0, 'num_object_copies': 15384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8152, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1808}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8126", 'reported_seq': 8247, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.484234+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:12:08.484234+0000', 'last_peered': '2024-09-15T03:12:08.484234+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:12:08.484234+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8126, 'log_dups_size': 0, 'ondisk_log_size': 8126, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31389298, 'num_objects': 3832, 'num_object_clones': 0, 'num_object_copies': 15328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8124, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1806}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:12:10.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8226", 'reported_seq': 8343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.496189+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:12:08.496189+0000', 'last_peered': '2024-09-15T03:12:08.496189+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:12:08.496189+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8226, 'log_dups_size': 0, 'ondisk_log_size': 8226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31610482, 'num_objects': 3859, 'num_object_clones': 0, 'num_object_copies': 15436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8224, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8127", 'reported_seq': 8218, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.622774+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:12:08.622774+0000', 'last_peered': '2024-09-15T03:12:08.622774+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:12:08.622774+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8127, 'log_dups_size': 0, 'ondisk_log_size': 8127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31476964, 'num_objects': 3843, 'num_object_clones': 0, 'num_object_copies': 15372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3843, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8124, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1816}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8092", 'reported_seq': 8206, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:03.650727+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:12:03.650727+0000', 'last_peered': '2024-09-15T03:12:03.650727+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:12:03.650727+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8092, 'log_dups_size': 0, 'ondisk_log_size': 8092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31749746, 'num_objects': 3876, 'num_object_clones': 0, 'num_object_copies': 15504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8090, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1834}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8136", 'reported_seq': 8230, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.463835+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:12:08.463835+0000', 'last_peered': '2024-09-15T03:12:08.463835+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:12:08.463835+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8136, 'log_dups_size': 0, 'ondisk_log_size': 8136, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30691755, 'num_objects': 3747, 'num_object_clones': 0, 'num_object_copies': 14988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8133, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1685}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8141", 'reported_seq': 8372, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:03.773074+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:12:03.773074+0000', 'last_peered': '2024-09-15T03:12:03.773074+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:12:03.773074+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8141, 'log_dups_size': 0, 'ondisk_log_size': 8141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32701241, 'num_objects': 3992, 'num_object_clones': 0, 'num_object_copies': 15968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8140, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 75, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2026}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:12:10.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'7975", 'reported_seq': 8075, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.485550+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:12:08.485550+0000', 'last_peered': '2024-09-15T03:12:08.485550+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:12:08.485550+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7975, 'log_dups_size': 0, 'ondisk_log_size': 7975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30365298, 'num_objects': 3707, 'num_object_clones': 0, 'num_object_copies': 14828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7973, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1694}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:10.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:12:10.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'7978", 'reported_seq': 8085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:03.618417+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:12:03.618417+0000', 'last_peered': '2024-09-15T03:12:03.618417+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:12:03.618417+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7978, 'log_dups_size': 0, 'ondisk_log_size': 7978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31987314, 'num_objects': 3905, 'num_object_clones': 0, 'num_object_copies': 15620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7976, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1928}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:10.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:12:10.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8167", 'reported_seq': 8278, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:08.516723+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:12:08.516723+0000', 'last_peered': '2024-09-15T03:12:08.516723+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:12:08.516723+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8167, 'log_dups_size': 0, 'ondisk_log_size': 8167, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31471218, 'num_objects': 3842, 'num_object_clones': 0, 'num_object_copies': 15368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8165, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:10.542 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:12:12.928 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:12.928 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28367/244164 objects degraded (11.618%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":455,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61041,"data_bytes":500005460,"bytes_used":2247299072,"bytes_avail":384299757568,"bytes_total":386547056640,"degraded_objects":28367,"degraded_total":244164,"degraded_ratio":0.11618010845169641,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":13000,"recovering_keys_per_sec":0,"num_objects_recovered":113,"num_bytes_recovered":139264,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":193},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:12:12.969 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:12:12.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:14.476 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:14.490 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:14.546 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-15T03:12:12.829456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":500005460,"num_objects":61041,"num_object_clones":0,"num_object_copies":244164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129666,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4317,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129693,"ondisk_log_size":129693,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2194628,"kb_used_data":978744,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375292732,"statfs":{"total":386547056640,"available":384299757568,"internally_reserved":0,"allocated":1002233856,"data_stored":1001535138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,2,10,12,8,12,8,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16998400,"num_objects":-2075,"num_object_clones":0,"num_object_copies":-8300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2075,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":139264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010985"},"pg_stats":[{"pgid":"1.f","version":"35'8046","reported_seq":8112,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.666881+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:08.666881+0000","last_peered":"2024-09-15T03:12:08.666881+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:08.666881+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31653888,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7810","reported_seq":7904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.555614+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:08.555614+0000","last_peered":"2024-09-15T03:12:08.555614+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:08.555614+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7810,"log_dups_size":0,"ondisk_log_size":7810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29619826,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8328","reported_seq":8434,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.723245+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:08.723245+0000","last_peered":"2024-09-15T03:12:08.723245+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:08.723245+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31882065,"num_objects":3893,"num_object_clones":0,"num_object_copies":15572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3893,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8327,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1801}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7865","reported_seq":11837,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:09.078069+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:09.078069+0000","last_peered":"2024-09-15T03:12:09.078069+0000","last_clean":"2024-09-15T03:12:09.078069+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:09.078069+0000","last_undegraded":"2024-09-15T03:12:09.078069+0000","last_fullsized":"2024-09-15T03:12:09.078069+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7865,"log_dups_size":0,"ondisk_log_size":7865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30824050,"num_objects":3763,"num_object_clones":0,"num_object_copies":15052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7863,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8246","reported_seq":8352,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.603056+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:08.603056+0000","last_peered":"2024-09-15T03:12:08.603056+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:08.603056+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8246,"log_dups_size":0,"ondisk_log_size":8246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31413874,"num_objects":3835,"num_object_clones":0,"num_object_copies":15340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8244,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1751}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8152","reported_seq":8236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.368854+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:08.368854+0000","last_peered":"2024-09-15T03:12:08.368854+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:08.368854+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8152,"log_dups_size":0,"ondisk_log_size":8152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31506432,"num_objects":3846,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8126","reported_seq":8247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.484234+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:08.484234+0000","last_peered":"2024-09-15T03:12:08.484234+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:08.484234+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8126,"log_dups_size":0,"ondisk_log_size":8126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31389298,"num_objects":3832,"num_object_clones":0,"num_object_copies":15328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1806}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8226","reported_seq":8343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.496189+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:08.496189+0000","last_peered":"2024-09-15T03:12:08.496189+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:08.496189+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8226,"log_dups_size":0,"ondisk_log_size":8226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31610482,"num_objects":3859,"num_object_clones":0,"num_object_copies":15436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8127","reported_seq":8218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.622774+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:08.622774+0000","last_peered":"2024-09-15T03:12:08.622774+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:08.622774+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31476964,"num_objects":3843,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8156","reported_seq":8275,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.625308+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:08.625308+0000","last_peered":"2024-09-15T03:12:08.625308+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:08.625308+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31225458,"num_objects":3812,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8136","reported_seq":8230,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.463835+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:08.463835+0000","last_peered":"2024-09-15T03:12:08.463835+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:08.463835+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30691755,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8061","reported_seq":12102,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:08.997327+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:08.997327+0000","last_peered":"2024-09-15T03:12:08.997327+0000","last_clean":"2024-09-15T03:12:08.997327+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:08.997327+0000","last_undegraded":"2024-09-15T03:12:08.997327+0000","last_fullsized":"2024-09-15T03:12:08.997327+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31439673,"num_objects":3838,"num_object_clones":0,"num_object_copies":15352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8060,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8219","reported_seq":8545,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.839598+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:08.839598+0000","last_peered":"2024-09-15T03:12:08.839598+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:08.839598+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8219,"log_dups_size":0,"ondisk_log_size":8219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32062265,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8218,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1996}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'7975","reported_seq":8075,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.485550+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:08.485550+0000","last_peered":"2024-09-15T03:12:08.485550+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:08.485550+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30365298,"num_objects":3707,"num_object_clones":0,"num_object_copies":14828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8053","reported_seq":8160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.622553+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:08.622553+0000","last_peered":"2024-09-15T03:12:08.622553+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:08.622553+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8053,"log_dups_size":0,"ondisk_log_size":8053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31372914,"num_objects":3830,"num_object_clones":0,"num_object_copies":15320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8051,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8167","reported_seq":8278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:08.516723+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:08.516723+0000","last_peered":"2024-09-15T03:12:08.516723+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:08.516723+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8167,"log_dups_size":0,"ondisk_log_size":8167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31471218,"num_objects":3842,"num_object_clones":0,"num_object_copies":15368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":500005460,"num_objects":61041,"num_object_clones":0,"num_object_copies":244164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61041,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129666,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4317,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":999788544,"data_stored":999788544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129693,"ondisk_log_size":129693,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542492,"kb_used_data":244624,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93829348,"statfs":{"total":96636764160,"available":96081252352,"internally_reserved":0,"allocated":250494976,"data_stored":250319510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,2,2,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.249}]},{"osd":1,"last update":"Sun Sep 15 03:11:13 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":2.2280000000000002,"15min":1.831},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.911999999999999,"5min":22.911999999999999,"15min":22.911999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.7549999999999999,"15min":1.371},"min":{"1min":0.25,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.798,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.38500000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546376,"kb_used_data":244860,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93825464,"statfs":{"total":96636764160,"available":96077275136,"internally_reserved":0,"allocated":250736640,"data_stored":250564320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3,4,2,4,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":6.1040000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.45900000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.40899999999999997},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.47699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.42399999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555128,"kb_used_data":244752,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93816712,"statfs":{"total":96636764160,"available":96068313088,"internally_reserved":0,"allocated":250626048,"data_stored":250450582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.1559999999999999,"15min":1.8959999999999999},"min":{"1min":0.19,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.8789999999999996,"5min":18.091000000000001,"15min":18.091000000000001},"last":9.6460000000000008},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.8440000000000001,"15min":1.579},"min":{"1min":0.23400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":12.244,"5min":20.791,"15min":20.791},"last":4.3239999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8519999999999999,"5min":1.871,"15min":1.5609999999999999},"min":{"1min":0.23599999999999999,"5min":0.219,"15min":0.219},"max":{"1min":26.911000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":3.113,"5min":1.9670000000000001,"15min":1.8160000000000001},"min":{"1min":0.221,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":20.055,"5min":20.055,"15min":20.055},"last":9.6829999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4380000000000002,"5min":1.671,"15min":1.663},"min":{"1min":0.22500000000000001,"5min":0.222,"15min":0.222},"max":{"1min":11.413,"5min":13.414999999999999,"15min":13.414999999999999},"last":9.3369999999999997},{"interface":"front","average":{"1min":2.2160000000000002,"5min":1.571,"15min":1.339},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":20.681000000000001,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.39900000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550632,"kb_used_data":244508,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93821208,"statfs":{"total":96636764160,"available":96072916992,"internally_reserved":0,"allocated":250376192,"data_stored":250200726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,0,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.50900000000000001},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.27900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":30.013000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249999360,"data_stored":249999360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249880576,"data_stored":249880576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249888768,"data_stored":249888768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250019840,"data_stored":250019840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:17.548 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:19.270 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:19.278 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:19.287 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-15T03:12:18.835122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":488487508,"num_objects":59635,"num_object_clones":0,"num_object_copies":238540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131072,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131099,"ondisk_log_size":131099,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2179232,"kb_used_data":959216,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375308128,"statfs":{"total":386547056640,"available":384315523072,"internally_reserved":0,"allocated":982237184,"data_stored":981538466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,6,8,13,15,9,6,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20119552,"num_objects":-2456,"num_object_clones":0,"num_object_copies":-9824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2456,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008995"},"pg_stats":[{"pgid":"1.f","version":"35'8170","reported_seq":8236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.781993+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:18.781993+0000","last_peered":"2024-09-15T03:12:18.781993+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:18.781993+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30638080,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8170,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7886","reported_seq":7980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.516580+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:13.516580+0000","last_peered":"2024-09-15T03:12:13.516580+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:13.516580+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7886,"log_dups_size":0,"ondisk_log_size":7886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28997234,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7884,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8390","reported_seq":8498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.747191+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:13.747191+0000","last_peered":"2024-09-15T03:12:13.747191+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:13.747191+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8390,"log_dups_size":0,"ondisk_log_size":8390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31374161,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8389,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1740}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7930","reported_seq":11902,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:13.811363+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:13.811363+0000","last_peered":"2024-09-15T03:12:13.811363+0000","last_clean":"2024-09-15T03:12:13.811363+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:13.811363+0000","last_undegraded":"2024-09-15T03:12:13.811363+0000","last_fullsized":"2024-09-15T03:12:13.811363+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7930,"log_dups_size":0,"ondisk_log_size":7930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291570,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7928,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8364","reported_seq":8470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.564482+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:18.564482+0000","last_peered":"2024-09-15T03:12:18.564482+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:18.564482+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30447218,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8362,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8282","reported_seq":8366,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.715250+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:18.715250+0000","last_peered":"2024-09-15T03:12:18.715250+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:18.715250+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30441472,"num_objects":3716,"num_object_clones":0,"num_object_copies":14864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8200","reported_seq":8318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.498705+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:13.498705+0000","last_peered":"2024-09-15T03:12:13.498705+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:13.498705+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30783090,"num_objects":3758,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1732}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8282","reported_seq":8399,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.627733+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:13.627733+0000","last_peered":"2024-09-15T03:12:13.627733+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:13.627733+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31151730,"num_objects":3803,"num_object_clones":0,"num_object_copies":15212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8264","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.557047+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:18.557047+0000","last_peered":"2024-09-15T03:12:18.557047+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:18.557047+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30354660,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8261,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8221","reported_seq":8340,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.771828+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:13.771828+0000","last_peered":"2024-09-15T03:12:13.771828+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:13.771828+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692978,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1705}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8262","reported_seq":8356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.658511+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:18.658511+0000","last_peered":"2024-09-15T03:12:18.658511+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:18.658511+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8262,"log_dups_size":0,"ondisk_log_size":8262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29659563,"num_objects":3621,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8115","reported_seq":12156,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:14.033035+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:14.033035+0000","last_peered":"2024-09-15T03:12:14.033035+0000","last_clean":"2024-09-15T03:12:14.033035+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:14.033035+0000","last_undegraded":"2024-09-15T03:12:14.033035+0000","last_fullsized":"2024-09-15T03:12:14.033035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8115,"log_dups_size":0,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30997305,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8114,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8284","reported_seq":8704,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.749364+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:13.749364+0000","last_peered":"2024-09-15T03:12:13.749364+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:13.749364+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529785,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8283,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1978}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8098","reported_seq":8198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.742411+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:18.742411+0000","last_peered":"2024-09-15T03:12:18.742411+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:18.742411+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8098,"log_dups_size":0,"ondisk_log_size":8098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29357682,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8096,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8117","reported_seq":8222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.850103+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:13.850103+0000","last_peered":"2024-09-15T03:12:13.850103+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:13.850103+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8117,"log_dups_size":0,"ondisk_log_size":8117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848626,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8115,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1789}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8234","reported_seq":8347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.625091+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:13.625091+0000","last_peered":"2024-09-15T03:12:13.625091+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:13.625091+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8234,"log_dups_size":0,"ondisk_log_size":8234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30922354,"num_objects":3775,"num_object_clones":0,"num_object_copies":15100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8232,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1716}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":488487508,"num_objects":59635,"num_object_clones":0,"num_object_copies":238540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131072,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":978591744,"data_stored":978591744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131099,"ondisk_log_size":131099,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542556,"kb_used_data":236356,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93829284,"statfs":{"total":96636764160,"available":96081186816,"internally_reserved":0,"allocated":242028544,"data_stored":241853078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,0,1,5,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.308}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":3.5579999999999998},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":3.8999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546520,"kb_used_data":240848,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93825320,"statfs":{"total":96636764160,"available":96077127680,"internally_reserved":0,"allocated":246628352,"data_stored":246456032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,2,4,2,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.71699999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.51900000000000002},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.93100000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538692,"kb_used_data":240828,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93833148,"statfs":{"total":96636764160,"available":96085143552,"internally_reserved":0,"allocated":246607872,"data_stored":246432406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.37},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.222}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.433},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":0.33400000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.47599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551464,"kb_used_data":241184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93820376,"statfs":{"total":96636764160,"available":96072065024,"internally_reserved":0,"allocated":246972416,"data_stored":246796950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.42799999999999999},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.32500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245809152,"data_stored":245809152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245608448,"data_stored":245608448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245719040,"data_stored":245719040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241455104,"data_stored":241455104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:19.288 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:20.810 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:20.826 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:20.865 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-15T03:12:18.835122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":488487508,"num_objects":59635,"num_object_clones":0,"num_object_copies":238540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131072,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131099,"ondisk_log_size":131099,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2179232,"kb_used_data":959216,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375308128,"statfs":{"total":386547056640,"available":384315523072,"internally_reserved":0,"allocated":982237184,"data_stored":981538466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,6,8,13,15,9,6,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20119552,"num_objects":-2456,"num_object_clones":0,"num_object_copies":-9824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2456,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008995"},"pg_stats":[{"pgid":"1.f","version":"35'8170","reported_seq":8236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.781993+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:18.781993+0000","last_peered":"2024-09-15T03:12:18.781993+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:18.781993+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30638080,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8170,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7886","reported_seq":7980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.516580+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:13.516580+0000","last_peered":"2024-09-15T03:12:13.516580+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:13.516580+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7886,"log_dups_size":0,"ondisk_log_size":7886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28997234,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7884,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1556}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8390","reported_seq":8498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.747191+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:13.747191+0000","last_peered":"2024-09-15T03:12:13.747191+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:13.747191+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8390,"log_dups_size":0,"ondisk_log_size":8390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31374161,"num_objects":3831,"num_object_clones":0,"num_object_copies":15324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8389,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1740}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7930","reported_seq":11902,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:13.811363+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:13.811363+0000","last_peered":"2024-09-15T03:12:13.811363+0000","last_clean":"2024-09-15T03:12:13.811363+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:13.811363+0000","last_undegraded":"2024-09-15T03:12:13.811363+0000","last_fullsized":"2024-09-15T03:12:13.811363+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7930,"log_dups_size":0,"ondisk_log_size":7930,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291570,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7928,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8364","reported_seq":8470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.564482+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:18.564482+0000","last_peered":"2024-09-15T03:12:18.564482+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:18.564482+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30447218,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8362,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8282","reported_seq":8366,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.715250+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:18.715250+0000","last_peered":"2024-09-15T03:12:18.715250+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:18.715250+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30441472,"num_objects":3716,"num_object_clones":0,"num_object_copies":14864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8200","reported_seq":8318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.498705+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:13.498705+0000","last_peered":"2024-09-15T03:12:13.498705+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:13.498705+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30783090,"num_objects":3758,"num_object_clones":0,"num_object_copies":15032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1732}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8282","reported_seq":8399,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.627733+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:13.627733+0000","last_peered":"2024-09-15T03:12:13.627733+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:13.627733+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31151730,"num_objects":3803,"num_object_clones":0,"num_object_copies":15212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8264","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.557047+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:18.557047+0000","last_peered":"2024-09-15T03:12:18.557047+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:18.557047+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30354660,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8261,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8221","reported_seq":8340,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.771828+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:13.771828+0000","last_peered":"2024-09-15T03:12:13.771828+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:13.771828+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8221,"log_dups_size":0,"ondisk_log_size":8221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692978,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8219,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1705}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8262","reported_seq":8356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.658511+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:18.658511+0000","last_peered":"2024-09-15T03:12:18.658511+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:18.658511+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8262,"log_dups_size":0,"ondisk_log_size":8262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29659563,"num_objects":3621,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8115","reported_seq":12156,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:14.033035+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:14.033035+0000","last_peered":"2024-09-15T03:12:14.033035+0000","last_clean":"2024-09-15T03:12:14.033035+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:14.033035+0000","last_undegraded":"2024-09-15T03:12:14.033035+0000","last_fullsized":"2024-09-15T03:12:14.033035+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8115,"log_dups_size":0,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30997305,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8114,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8284","reported_seq":8704,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.749364+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:13.749364+0000","last_peered":"2024-09-15T03:12:13.749364+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:13.749364+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8284,"log_dups_size":0,"ondisk_log_size":8284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31529785,"num_objects":3849,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1871,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8283,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1978}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8098","reported_seq":8198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.742411+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:18.742411+0000","last_peered":"2024-09-15T03:12:18.742411+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:18.742411+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8098,"log_dups_size":0,"ondisk_log_size":8098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29357682,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8096,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8117","reported_seq":8222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.850103+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:13.850103+0000","last_peered":"2024-09-15T03:12:13.850103+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:13.850103+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8117,"log_dups_size":0,"ondisk_log_size":8117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30848626,"num_objects":3766,"num_object_clones":0,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8115,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1789}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8234","reported_seq":8347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:13.625091+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:13.625091+0000","last_peered":"2024-09-15T03:12:13.625091+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:13.625091+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8234,"log_dups_size":0,"ondisk_log_size":8234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30922354,"num_objects":3775,"num_object_clones":0,"num_object_copies":15100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8232,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1716}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":488487508,"num_objects":59635,"num_object_clones":0,"num_object_copies":238540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131072,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4365,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":978591744,"data_stored":978591744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131099,"ondisk_log_size":131099,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542556,"kb_used_data":236356,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93829284,"statfs":{"total":96636764160,"available":96081186816,"internally_reserved":0,"allocated":242028544,"data_stored":241853078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,0,1,5,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.308}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":3.5579999999999998},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":3.8999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546520,"kb_used_data":240848,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93825320,"statfs":{"total":96636764160,"available":96077127680,"internally_reserved":0,"allocated":246628352,"data_stored":246456032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,2,4,2,1,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.71699999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.534,"15min":1.214},"min":{"1min":0.314,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":14.632999999999999,"5min":19.343,"15min":19.343},"last":0.51900000000000002},{"interface":"front","average":{"1min":2.6859999999999999,"5min":1.625,"15min":1.2669999999999999},"min":{"1min":0.36899999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":14.675000000000001,"5min":23.311,"15min":23.311},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.93100000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538692,"kb_used_data":240828,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93833148,"statfs":{"total":96636764160,"available":96085143552,"internally_reserved":0,"allocated":246607872,"data_stored":246432406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.37},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.222}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.433},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":0.33400000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.47599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551464,"kb_used_data":241184,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93820376,"statfs":{"total":96636764160,"available":96072065024,"internally_reserved":0,"allocated":246972416,"data_stored":246796950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.42799999999999999},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.32500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.39900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245809152,"data_stored":245809152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245608448,"data_stored":245608448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":245719040,"data_stored":245719040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241455104,"data_stored":241455104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:20.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:12:20.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8170", 'reported_seq': 8236, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:18.781993+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:12:18.781993+0000', 'last_peered': '2024-09-15T03:12:18.781993+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:12:18.781993+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8170, 'log_dups_size': 0, 'ondisk_log_size': 8170, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30638080, 'num_objects': 3740, 'num_object_clones': 0, 'num_object_copies': 14960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8170, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1774}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:20.866 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:12:20.866 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'7886", 'reported_seq': 7980, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:13.516580+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:12:13.516580+0000', 'last_peered': '2024-09-15T03:12:13.516580+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:12:13.516580+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 7886, 'log_dups_size': 0, 'ondisk_log_size': 7886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28997234, 'num_objects': 3540, 'num_object_clones': 0, 'num_object_copies': 14160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7884, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1556}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8390", 'reported_seq': 8498, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:13.747191+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:12:13.747191+0000', 'last_peered': '2024-09-15T03:12:13.747191+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:12:13.747191+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8390, 'log_dups_size': 0, 'ondisk_log_size': 8390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31374161, 'num_objects': 3831, 'num_object_clones': 0, 'num_object_copies': 15324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3831, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8389, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1740}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8364", 'reported_seq': 8470, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:18.564482+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:12:18.564482+0000', 'last_peered': '2024-09-15T03:12:18.564482+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:12:18.564482+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8364, 'log_dups_size': 0, 'ondisk_log_size': 8364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30447218, 'num_objects': 3717, 'num_object_clones': 0, 'num_object_copies': 14868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8362, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1633}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8282", 'reported_seq': 8366, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:18.715250+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:12:18.715250+0000', 'last_peered': '2024-09-15T03:12:18.715250+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:12:18.715250+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8282, 'log_dups_size': 0, 'ondisk_log_size': 8282, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30441472, 'num_objects': 3716, 'num_object_clones': 0, 'num_object_copies': 14864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8282, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1678}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8200", 'reported_seq': 8318, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:13.498705+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:12:13.498705+0000', 'last_peered': '2024-09-15T03:12:13.498705+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:12:13.498705+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8200, 'log_dups_size': 0, 'ondisk_log_size': 8200, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30783090, 'num_objects': 3758, 'num_object_clones': 0, 'num_object_copies': 15032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8198, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1732}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8282", 'reported_seq': 8399, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:13.627733+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:12:13.627733+0000', 'last_peered': '2024-09-15T03:12:13.627733+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:12:13.627733+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8282, 'log_dups_size': 0, 'ondisk_log_size': 8282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31151730, 'num_objects': 3803, 'num_object_clones': 0, 'num_object_copies': 15212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8280, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:20.867 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8264", 'reported_seq': 8355, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:18.557047+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:12:18.557047+0000', 'last_peered': '2024-09-15T03:12:18.557047+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:12:18.557047+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8264, 'log_dups_size': 0, 'ondisk_log_size': 8264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30354660, 'num_objects': 3706, 'num_object_clones': 0, 'num_object_copies': 14824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8261, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1679}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8221", 'reported_seq': 8340, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:13.771828+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:12:13.771828+0000', 'last_peered': '2024-09-15T03:12:13.771828+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:12:13.771828+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8221, 'log_dups_size': 0, 'ondisk_log_size': 8221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30692978, 'num_objects': 3747, 'num_object_clones': 0, 'num_object_copies': 14988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8219, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1705}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8262", 'reported_seq': 8356, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:18.658511+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:12:18.658511+0000', 'last_peered': '2024-09-15T03:12:18.658511+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:12:18.658511+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8262, 'log_dups_size': 0, 'ondisk_log_size': 8262, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29659563, 'num_objects': 3621, 'num_object_clones': 0, 'num_object_copies': 14484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8259, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1559}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8284", 'reported_seq': 8704, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:13.749364+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:12:13.749364+0000', 'last_peered': '2024-09-15T03:12:13.749364+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:12:13.749364+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8284, 'log_dups_size': 0, 'ondisk_log_size': 8284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31529785, 'num_objects': 3849, 'num_object_clones': 0, 'num_object_copies': 15396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1871, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8283, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1978}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8098", 'reported_seq': 8198, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:18.742411+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:12:18.742411+0000', 'last_peered': '2024-09-15T03:12:18.742411+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:12:18.742411+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8098, 'log_dups_size': 0, 'ondisk_log_size': 8098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29357682, 'num_objects': 3584, 'num_object_clones': 0, 'num_object_copies': 14336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8096, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1571}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:12:20.868 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8117", 'reported_seq': 8222, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:13.850103+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:12:13.850103+0000', 'last_peered': '2024-09-15T03:12:13.850103+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:12:13.850103+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8117, 'log_dups_size': 0, 'ondisk_log_size': 8117, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30848626, 'num_objects': 3766, 'num_object_clones': 0, 'num_object_copies': 15064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3766, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8115, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1789}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:20.869 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:12:20.869 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8234", 'reported_seq': 8347, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:13.625091+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:12:13.625091+0000', 'last_peered': '2024-09-15T03:12:13.625091+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:12:13.625091+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8234, 'log_dups_size': 0, 'ondisk_log_size': 8234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30922354, 'num_objects': 3775, 'num_object_clones': 0, 'num_object_copies': 15100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3775, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8232, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1716}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:20.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:12:22.830 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:22.830 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28263/235792 objects degraded (11.986%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":465,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58948,"data_bytes":482859604,"bytes_used":2217656320,"bytes_avail":384329400320,"bytes_total":386547056640,"degraded_objects":28263,"degraded_total":235792,"degraded_ratio":0.11986411752731221,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":209},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:12:22.854 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:12:22.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:24.793 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:24.854 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:24.887 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-15T03:12:22.836133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":482859604,"num_objects":58948,"num_object_clones":0,"num_object_copies":235792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131759,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4421,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131786,"ondisk_log_size":131786,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2165680,"kb_used_data":945664,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375321680,"statfs":{"total":386547056640,"available":384329400320,"internally_reserved":0,"allocated":968359936,"data_stored":967661218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,5,6,11,9,12,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17145856,"num_objects":-2093,"num_object_clones":0,"num_object_copies":-8372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2093,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007177"},"pg_stats":[{"pgid":"1.f","version":"35'8170","reported_seq":8236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.781993+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:18.781993+0000","last_peered":"2024-09-15T03:12:18.781993+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:18.781993+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30638080,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8170,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1774}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'7965","reported_seq":8059,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.711497+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:18.711497+0000","last_peered":"2024-09-15T03:12:18.711497+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:18.711497+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7965,"log_dups_size":0,"ondisk_log_size":7965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28350066,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7963,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8468","reported_seq":8576,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.799512+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:18.799512+0000","last_peered":"2024-09-15T03:12:18.799512+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:18.799512+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8468,"log_dups_size":0,"ondisk_log_size":8468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30735185,"num_objects":3753,"num_object_clones":0,"num_object_copies":15012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3753,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8467,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1662}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'7997","reported_seq":11969,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:19.153793+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:19.153793+0000","last_peered":"2024-09-15T03:12:19.153793+0000","last_clean":"2024-09-15T03:12:19.153793+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:19.153793+0000","last_undegraded":"2024-09-15T03:12:19.153793+0000","last_fullsized":"2024-09-15T03:12:19.153793+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":7997,"log_dups_size":0,"ondisk_log_size":7997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29742706,"num_objects":3631,"num_object_clones":0,"num_object_copies":14524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7995,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8364","reported_seq":8470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.564482+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:18.564482+0000","last_peered":"2024-09-15T03:12:18.564482+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:18.564482+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30447218,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8362,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8282","reported_seq":8366,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.715250+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:18.715250+0000","last_peered":"2024-09-15T03:12:18.715250+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:18.715250+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30441472,"num_objects":3716,"num_object_clones":0,"num_object_copies":14864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8282,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1678}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8271","reported_seq":8392,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.670001+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:18.670001+0000","last_peered":"2024-09-15T03:12:18.670001+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:18.670001+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8271,"log_dups_size":0,"ondisk_log_size":8271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30201458,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8269,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1661}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8347","reported_seq":8464,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.646731+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:18.646731+0000","last_peered":"2024-09-15T03:12:18.646731+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:18.646731+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30619250,"num_objects":3738,"num_object_clones":0,"num_object_copies":14952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8345,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8264","reported_seq":8355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.557047+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:18.557047+0000","last_peered":"2024-09-15T03:12:18.557047+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:18.557047+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30354660,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8261,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8285","reported_seq":8404,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.649012+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:18.649012+0000","last_peered":"2024-09-15T03:12:18.649012+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:18.649012+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8285,"log_dups_size":0,"ondisk_log_size":8285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30168690,"num_objects":3683,"num_object_clones":0,"num_object_copies":14732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8283,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1641}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8262","reported_seq":8356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.658511+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:18.658511+0000","last_peered":"2024-09-15T03:12:18.658511+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:18.658511+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8262,"log_dups_size":0,"ondisk_log_size":8262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29659563,"num_objects":3621,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8189","reported_seq":12230,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:19.250257+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:19.250257+0000","last_peered":"2024-09-15T03:12:19.250257+0000","last_clean":"2024-09-15T03:12:19.250257+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:19.250257+0000","last_undegraded":"2024-09-15T03:12:19.250257+0000","last_fullsized":"2024-09-15T03:12:19.250257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30391097,"num_objects":3710,"num_object_clones":0,"num_object_copies":14840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8188,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8347","reported_seq":8879,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.836778+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:18.836778+0000","last_peered":"2024-09-15T03:12:18.836778+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:18.836778+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31013689,"num_objects":3786,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8346,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1971}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8098","reported_seq":8198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.742411+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:18.742411+0000","last_peered":"2024-09-15T03:12:18.742411+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:18.742411+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8098,"log_dups_size":0,"ondisk_log_size":8098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29357682,"num_objects":3584,"num_object_clones":0,"num_object_copies":14336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8096,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8185","reported_seq":8292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.551625+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:18.551625+0000","last_peered":"2024-09-15T03:12:18.551625+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:18.551625+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8185,"log_dups_size":0,"ondisk_log_size":8185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30291570,"num_objects":3698,"num_object_clones":0,"num_object_copies":14792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8183,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1721}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8292","reported_seq":8405,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:18.547382+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:18.547382+0000","last_peered":"2024-09-15T03:12:18.547382+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:18.547382+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8292,"log_dups_size":0,"ondisk_log_size":8292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30447218,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8290,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":482859604,"num_objects":58948,"num_object_clones":0,"num_object_copies":235792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131759,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4421,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":965464064,"data_stored":965464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131786,"ondisk_log_size":131786,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542556,"kb_used_data":236356,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93829284,"statfs":{"total":96636764160,"available":96081186816,"internally_reserved":0,"allocated":242028544,"data_stored":241853078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1,0,1,5,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.308}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":3.5579999999999998},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":3.8999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550620,"kb_used_data":236628,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93821220,"statfs":{"total":96636764160,"available":96072929280,"internally_reserved":0,"allocated":242307072,"data_stored":242134752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":14.653},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":4.4660000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.35299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":12.031000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538596,"kb_used_data":236572,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93833244,"statfs":{"total":96636764160,"available":96085241856,"internally_reserved":0,"allocated":242249728,"data_stored":242074262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":6.5430000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":8.1059999999999999},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":0.47999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":8.0739999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533908,"kb_used_data":236108,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93837932,"statfs":{"total":96636764160,"available":96090042368,"internally_reserved":0,"allocated":241774592,"data_stored":241599126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:11:17 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":1.7410000000000001,"15min":1.0820000000000001},"min":{"1min":0.19700000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":8.7929999999999993,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":3.1019999999999999,"5min":2.7519999999999998,"15min":1.9299999999999999},"min":{"1min":0.25600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":9.9969999999999999,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.38200000000000001},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.214}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.252},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.33000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241410048,"data_stored":241410048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241238016,"data_stored":241238016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241360896,"data_stored":241360896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":241455104,"data_stored":241455104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:27.890 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:29.409 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:29.418 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:29.428 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-15T03:12:28.840476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":474552916,"num_objects":57934,"num_object_clones":0,"num_object_copies":231736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132773,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4484,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132800,"ondisk_log_size":132800,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2158420,"kb_used_data":930088,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375328940,"statfs":{"total":386547056640,"available":384336834560,"internally_reserved":0,"allocated":952410112,"data_stored":951711394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,3,6,11,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17047552,"num_objects":-2081,"num_object_clones":0,"num_object_copies":-8324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005822"},"pg_stats":[{"pgid":"1.f","version":"35'8237","reported_seq":8303,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.788645+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:23.788645+0000","last_peered":"2024-09-15T03:12:23.788645+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:23.788645+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8237,"log_dups_size":0,"ondisk_log_size":8237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30089216,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8237,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1707}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8033","reported_seq":8127,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.517942+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:23.517942+0000","last_peered":"2024-09-15T03:12:23.517942+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:23.517942+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8033,"log_dups_size":0,"ondisk_log_size":8033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27793010,"num_objects":3393,"num_object_clones":0,"num_object_copies":13572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8031,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8528","reported_seq":8636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.896019+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:23.896019+0000","last_peered":"2024-09-15T03:12:23.896019+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:23.896019+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8528,"log_dups_size":0,"ondisk_log_size":8528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30243665,"num_objects":3693,"num_object_clones":0,"num_object_copies":14772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3693,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8527,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1602}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8043","reported_seq":12015,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:24.289199+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:24.289199+0000","last_peered":"2024-09-15T03:12:24.289199+0000","last_clean":"2024-09-15T03:12:24.289199+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:24.289199+0000","last_undegraded":"2024-09-15T03:12:24.289199+0000","last_fullsized":"2024-09-15T03:12:24.289199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8043,"log_dups_size":0,"ondisk_log_size":8043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29365874,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8041,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8424","reported_seq":8527,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.709248+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:23.709248+0000","last_peered":"2024-09-15T03:12:23.709248+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:23.709248+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955698,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8351","reported_seq":8435,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.771269+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:23.771269+0000","last_peered":"2024-09-15T03:12:23.771269+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:23.771269+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29876224,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8351,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8336","reported_seq":8454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.716577+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:23.716577+0000","last_peered":"2024-09-15T03:12:23.716577+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:23.716577+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8336,"log_dups_size":0,"ondisk_log_size":8336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29668978,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8334,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1596}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8413","reported_seq":8530,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.687120+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:23.687120+0000","last_peered":"2024-09-15T03:12:23.687120+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:23.687120+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8413,"log_dups_size":0,"ondisk_log_size":8413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30078578,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8411,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1616}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8332","reported_seq":8423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.484950+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:23.484950+0000","last_peered":"2024-09-15T03:12:23.484950+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:23.484950+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8332,"log_dups_size":0,"ondisk_log_size":8332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29797604,"num_objects":3638,"num_object_clones":0,"num_object_copies":14552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8329,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1611}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8353","reported_seq":8472,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.768667+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:23.768667+0000","last_peered":"2024-09-15T03:12:23.768667+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:23.768667+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8353,"log_dups_size":0,"ondisk_log_size":8353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29611634,"num_objects":3615,"num_object_clones":0,"num_object_copies":14460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8351,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1573}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8338","reported_seq":8432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.760818+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:23.760818+0000","last_peered":"2024-09-15T03:12:23.760818+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:23.760818+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8338,"log_dups_size":0,"ondisk_log_size":8338,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29036971,"num_objects":3545,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8335,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8244","reported_seq":12285,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:24.340624+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:24.340624+0000","last_peered":"2024-09-15T03:12:24.340624+0000","last_clean":"2024-09-15T03:12:24.340624+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:24.340624+0000","last_undegraded":"2024-09-15T03:12:24.340624+0000","last_fullsized":"2024-09-15T03:12:24.340624+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29940537,"num_objects":3655,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8400","reported_seq":9057,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.956961+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:23.956961+0000","last_peered":"2024-09-15T03:12:23.956961+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:23.956961+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8400,"log_dups_size":0,"ondisk_log_size":8400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579513,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1752,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8399,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":289,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1981}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8163","reported_seq":8259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.581506+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:23.581506+0000","last_peered":"2024-09-15T03:12:23.581506+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:23.581506+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8163,"log_dups_size":0,"ondisk_log_size":8163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28825202,"num_objects":3519,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8161,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1506}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8251","reported_seq":8358,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.748120+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:23.748120+0000","last_peered":"2024-09-15T03:12:23.748120+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:23.748120+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8251,"log_dups_size":0,"ondisk_log_size":8251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29750898,"num_objects":3632,"num_object_clones":0,"num_object_copies":14528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8249,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1655}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8354","reported_seq":8467,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:23.723932+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:23.723932+0000","last_peered":"2024-09-15T03:12:23.723932+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:23.723932+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8354,"log_dups_size":0,"ondisk_log_size":8354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29939314,"num_objects":3655,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1596}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":474552916,"num_objects":57934,"num_object_clones":0,"num_object_copies":231736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132773,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4484,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":948801536,"data_stored":948801536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132800,"ondisk_log_size":132800,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":546832,"kb_used_data":232328,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93825008,"statfs":{"total":96636764160,"available":96076808192,"internally_reserved":0,"allocated":237903872,"data_stored":237728406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.183},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.26100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":2.282},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.51100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534020,"kb_used_data":232496,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93837820,"statfs":{"total":96636764160,"available":96089927680,"internally_reserved":0,"allocated":238075904,"data_stored":237903584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,5,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.45500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.432},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.66300000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542804,"kb_used_data":232460,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93829036,"statfs":{"total":96636764160,"available":96080932864,"internally_reserved":0,"allocated":238039040,"data_stored":237863574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":2.6520000000000001},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":4.1829999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":0.25800000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.22800000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534764,"kb_used_data":232804,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93837076,"statfs":{"total":96636764160,"available":96089165824,"internally_reserved":0,"allocated":238391296,"data_stored":238215830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.7679999999999998},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":3.9510000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":4.0039999999999996},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.60699999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237277184,"data_stored":237277184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237027328,"data_stored":237027328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237187072,"data_stored":237187072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":237309952,"data_stored":237309952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:29.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:30.996 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:31.010 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:31.032 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-15T03:12:30.840941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":466033236,"num_objects":56894,"num_object_clones":0,"num_object_copies":227576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133813,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133840,"ondisk_log_size":133840,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2128656,"kb_used_data":912800,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375358704,"statfs":{"total":386547056640,"available":384367312896,"internally_reserved":0,"allocated":934707200,"data_stored":934008482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,10,10,16,19,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22454272,"num_objects":-2741,"num_object_clones":0,"num_object_copies":-10964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2741,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005818"},"pg_stats":[{"pgid":"1.f","version":"35'8280","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.750677+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:28.750677+0000","last_peered":"2024-09-15T03:12:28.750677+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:28.750677+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8109","reported_seq":8203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.747802+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:28.747802+0000","last_peered":"2024-09-15T03:12:28.747802+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:28.747802+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8109,"log_dups_size":0,"ondisk_log_size":8109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27170418,"num_objects":3317,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8600","reported_seq":8707,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.988303+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:28.988303+0000","last_peered":"2024-09-15T03:12:28.988303+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:28.988303+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29653841,"num_objects":3621,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8599,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8103","reported_seq":12075,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:29.350713+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:29.350713+0000","last_peered":"2024-09-15T03:12:29.350713+0000","last_clean":"2024-09-15T03:12:29.350713+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:29.350713+0000","last_undegraded":"2024-09-15T03:12:29.350713+0000","last_fullsized":"2024-09-15T03:12:29.350713+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8103,"log_dups_size":0,"ondisk_log_size":8103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28874354,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8101,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8482","reported_seq":8587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.719633+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:28.719633+0000","last_peered":"2024-09-15T03:12:28.719633+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:28.719633+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8482,"log_dups_size":0,"ondisk_log_size":8482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29480562,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8480,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8413","reported_seq":8497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.889983+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:28.889983+0000","last_peered":"2024-09-15T03:12:28.889983+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:28.889983+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8413,"log_dups_size":0,"ondisk_log_size":8413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29368320,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8403","reported_seq":8522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.697842+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:28.697842+0000","last_peered":"2024-09-15T03:12:28.697842+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:28.697842+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29120114,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8401,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8488","reported_seq":8604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.702278+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:28.702278+0000","last_peered":"2024-09-15T03:12:28.702278+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:28.702278+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29464178,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8486,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8403","reported_seq":8494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.677093+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:28.677093+0000","last_peered":"2024-09-15T03:12:28.677093+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:28.677093+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29215972,"num_objects":3567,"num_object_clones":0,"num_object_copies":14268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8415","reported_seq":8534,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.909065+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:28.909065+0000","last_peered":"2024-09-15T03:12:28.909065+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:28.909065+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8415,"log_dups_size":0,"ondisk_log_size":8415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29103730,"num_objects":3553,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8400","reported_seq":8494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.684857+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:28.684857+0000","last_peered":"2024-09-15T03:12:28.684857+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:28.684857+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8400,"log_dups_size":0,"ondisk_log_size":8400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28529067,"num_objects":3483,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8397,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8314","reported_seq":12355,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:29.381120+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:29.381120+0000","last_peered":"2024-09-15T03:12:29.381120+0000","last_clean":"2024-09-15T03:12:29.381120+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:29.381120+0000","last_undegraded":"2024-09-15T03:12:29.381120+0000","last_fullsized":"2024-09-15T03:12:29.381120+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8314,"log_dups_size":0,"ondisk_log_size":8314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29367097,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8313,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8470","reported_seq":9234,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.950032+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:28.950032+0000","last_peered":"2024-09-15T03:12:28.950032+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:28.950032+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8470,"log_dups_size":0,"ondisk_log_size":8470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30006073,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1964}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8229","reported_seq":8329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.785969+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:28.785969+0000","last_peered":"2024-09-15T03:12:28.785969+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:28.785969+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8229,"log_dups_size":0,"ondisk_log_size":8229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28284530,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8315","reported_seq":8422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:29.017804+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:29.017804+0000","last_peered":"2024-09-15T03:12:29.017804+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:29.017804+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29226610,"num_objects":3568,"num_object_clones":0,"num_object_copies":14272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8313,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1591}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8416","reported_seq":8529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.705138+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:28.705138+0000","last_peered":"2024-09-15T03:12:28.705138+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:28.705138+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29431410,"num_objects":3593,"num_object_clones":0,"num_object_copies":14372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8414,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":466033236,"num_objects":56894,"num_object_clones":0,"num_object_copies":227576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133813,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":931844096,"data_stored":931844096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133840,"ondisk_log_size":133840,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530264,"kb_used_data":228240,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93841576,"statfs":{"total":96636764160,"available":96093773824,"internally_reserved":0,"allocated":233717760,"data_stored":233542294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":4.2960000000000003},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.75600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":4.3630000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.88400000000000001},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":4.3360000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533996,"kb_used_data":228324,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93837844,"statfs":{"total":96636764160,"available":96089952256,"internally_reserved":0,"allocated":233803776,"data_stored":233631456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,1,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":2.351}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.84799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.376}]}]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526196,"kb_used_data":228316,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93845644,"statfs":{"total":96636764160,"available":96097939456,"internally_reserved":0,"allocated":233795584,"data_stored":233620118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,4,4,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.17399999999999999},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.318}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.38},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":0.20899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.33800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538200,"kb_used_data":227920,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93833640,"statfs":{"total":96636764160,"available":96085647360,"internally_reserved":0,"allocated":233390080,"data_stored":233214614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.314},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233046016,"data_stored":233046016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":232808448,"data_stored":232808448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232935424,"data_stored":232935424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233054208,"data_stored":233054208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:31.033 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8280", 'reported_seq': 8346, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.750677+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:12:28.750677+0000', 'last_peered': '2024-09-15T03:12:28.750677+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:12:28.750677+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8280, 'log_dups_size': 0, 'ondisk_log_size': 8280, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29736960, 'num_objects': 3630, 'num_object_clones': 0, 'num_object_copies': 14520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8280, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1664}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8109", 'reported_seq': 8203, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.747802+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:12:28.747802+0000', 'last_peered': '2024-09-15T03:12:28.747802+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:12:28.747802+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8109, 'log_dups_size': 0, 'ondisk_log_size': 8109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27170418, 'num_objects': 3317, 'num_object_clones': 0, 'num_object_copies': 13268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8107, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8600", 'reported_seq': 8707, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.988303+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:12:28.988303+0000', 'last_peered': '2024-09-15T03:12:28.988303+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:12:28.988303+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8600, 'log_dups_size': 0, 'ondisk_log_size': 8600, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29653841, 'num_objects': 3621, 'num_object_clones': 0, 'num_object_copies': 14484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3621, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8599, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1530}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8482", 'reported_seq': 8587, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.719633+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:12:28.719633+0000', 'last_peered': '2024-09-15T03:12:28.719633+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:12:28.719633+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8482, 'log_dups_size': 0, 'ondisk_log_size': 8482, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29480562, 'num_objects': 3599, 'num_object_clones': 0, 'num_object_copies': 14396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8480, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1515}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:12:31.034 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8413", 'reported_seq': 8497, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.889983+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:12:28.889983+0000', 'last_peered': '2024-09-15T03:12:28.889983+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:12:28.889983+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8413, 'log_dups_size': 0, 'ondisk_log_size': 8413, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29368320, 'num_objects': 3585, 'num_object_clones': 0, 'num_object_copies': 14340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8413, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1547}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8403", 'reported_seq': 8522, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.697842+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:12:28.697842+0000', 'last_peered': '2024-09-15T03:12:28.697842+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:12:28.697842+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8403, 'log_dups_size': 0, 'ondisk_log_size': 8403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29120114, 'num_objects': 3555, 'num_object_clones': 0, 'num_object_copies': 14220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8401, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1529}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8488", 'reported_seq': 8604, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.702278+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:12:28.702278+0000', 'last_peered': '2024-09-15T03:12:28.702278+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:12:28.702278+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8488, 'log_dups_size': 0, 'ondisk_log_size': 8488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29464178, 'num_objects': 3597, 'num_object_clones': 0, 'num_object_copies': 14388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8486, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1541}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8403", 'reported_seq': 8494, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.677093+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:12:28.677093+0000', 'last_peered': '2024-09-15T03:12:28.677093+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:12:28.677093+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8403, 'log_dups_size': 0, 'ondisk_log_size': 8403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29215972, 'num_objects': 3567, 'num_object_clones': 0, 'num_object_copies': 14268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3567, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8400, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1540}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8415", 'reported_seq': 8534, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.909065+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:12:28.909065+0000', 'last_peered': '2024-09-15T03:12:28.909065+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:12:28.909065+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8415, 'log_dups_size': 0, 'ondisk_log_size': 8415, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29103730, 'num_objects': 3553, 'num_object_clones': 0, 'num_object_copies': 14212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8413, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1511}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8400", 'reported_seq': 8494, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.684857+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:12:28.684857+0000', 'last_peered': '2024-09-15T03:12:28.684857+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:12:28.684857+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8400, 'log_dups_size': 0, 'ondisk_log_size': 8400, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28529067, 'num_objects': 3483, 'num_object_clones': 0, 'num_object_copies': 13932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8397, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1421}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:31.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:12:31.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8470", 'reported_seq': 9234, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.950032+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:12:28.950032+0000', 'last_peered': '2024-09-15T03:12:28.950032+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:12:28.950032+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8470, 'log_dups_size': 0, 'ondisk_log_size': 8470, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30006073, 'num_objects': 3663, 'num_object_clones': 0, 'num_object_copies': 14652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1699, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8469, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 342, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1964}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:31.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:12:31.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8229", 'reported_seq': 8329, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.785969+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:12:28.785969+0000', 'last_peered': '2024-09-15T03:12:28.785969+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:12:28.785969+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8229, 'log_dups_size': 0, 'ondisk_log_size': 8229, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28284530, 'num_objects': 3453, 'num_object_clones': 0, 'num_object_copies': 13812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8227, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1440}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:31.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:12:31.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8315", 'reported_seq': 8422, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:29.017804+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:12:29.017804+0000', 'last_peered': '2024-09-15T03:12:29.017804+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:12:29.017804+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8315, 'log_dups_size': 0, 'ondisk_log_size': 8315, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29226610, 'num_objects': 3568, 'num_object_clones': 0, 'num_object_copies': 14272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8313, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1591}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:31.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:12:31.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8416", 'reported_seq': 8529, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:28.705138+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:12:28.705138+0000', 'last_peered': '2024-09-15T03:12:28.705138+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:12:28.705138+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8416, 'log_dups_size': 0, 'ondisk_log_size': 8416, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29431410, 'num_objects': 3593, 'num_object_clones': 0, 'num_object_copies': 14372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8414, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:31.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:12:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:32.957 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28147/227576 objects degraded (12.368%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":476,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56894,"data_bytes":466033236,"bytes_used":2179743744,"bytes_avail":384367312896,"bytes_total":386547056640,"degraded_objects":28147,"degraded_total":227576,"degraded_ratio":0.12368175906070938,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":228},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:12:33.002 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:12:33.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:34.633 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:34.694 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:34.701 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-15T03:12:32.841446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":466033236,"num_objects":56894,"num_object_clones":0,"num_object_copies":227576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133813,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133840,"ondisk_log_size":133840,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2128656,"kb_used_data":912800,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375358704,"statfs":{"total":386547056640,"available":384367312896,"internally_reserved":0,"allocated":934707200,"data_stored":934008482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,10,10,16,19,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16826368,"num_objects":-2054,"num_object_clones":0,"num_object_copies":-8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005810"},"pg_stats":[{"pgid":"1.f","version":"35'8280","reported_seq":8346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.750677+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:28.750677+0000","last_peered":"2024-09-15T03:12:28.750677+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:28.750677+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8280,"log_dups_size":0,"ondisk_log_size":8280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29736960,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8109","reported_seq":8203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.747802+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:28.747802+0000","last_peered":"2024-09-15T03:12:28.747802+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:28.747802+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8109,"log_dups_size":0,"ondisk_log_size":8109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27170418,"num_objects":3317,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1333}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8600","reported_seq":8707,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.988303+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:28.988303+0000","last_peered":"2024-09-15T03:12:28.988303+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:28.988303+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8600,"log_dups_size":0,"ondisk_log_size":8600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29653841,"num_objects":3621,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8599,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8103","reported_seq":12075,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:29.350713+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:29.350713+0000","last_peered":"2024-09-15T03:12:29.350713+0000","last_clean":"2024-09-15T03:12:29.350713+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:29.350713+0000","last_undegraded":"2024-09-15T03:12:29.350713+0000","last_fullsized":"2024-09-15T03:12:29.350713+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8103,"log_dups_size":0,"ondisk_log_size":8103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28874354,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8101,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8482","reported_seq":8587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.719633+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:28.719633+0000","last_peered":"2024-09-15T03:12:28.719633+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:28.719633+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8482,"log_dups_size":0,"ondisk_log_size":8482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29480562,"num_objects":3599,"num_object_clones":0,"num_object_copies":14396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8480,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1515}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8413","reported_seq":8497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.889983+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:28.889983+0000","last_peered":"2024-09-15T03:12:28.889983+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:28.889983+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8413,"log_dups_size":0,"ondisk_log_size":8413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29368320,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1547}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8403","reported_seq":8522,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.697842+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:28.697842+0000","last_peered":"2024-09-15T03:12:28.697842+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:28.697842+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29120114,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8401,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1529}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8488","reported_seq":8604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.702278+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:28.702278+0000","last_peered":"2024-09-15T03:12:28.702278+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:28.702278+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8488,"log_dups_size":0,"ondisk_log_size":8488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29464178,"num_objects":3597,"num_object_clones":0,"num_object_copies":14388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8486,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1541}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8403","reported_seq":8494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.677093+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:28.677093+0000","last_peered":"2024-09-15T03:12:28.677093+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:28.677093+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29215972,"num_objects":3567,"num_object_clones":0,"num_object_copies":14268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1540}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8415","reported_seq":8534,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.909065+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:28.909065+0000","last_peered":"2024-09-15T03:12:28.909065+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:28.909065+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8415,"log_dups_size":0,"ondisk_log_size":8415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29103730,"num_objects":3553,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8400","reported_seq":8494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.684857+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:28.684857+0000","last_peered":"2024-09-15T03:12:28.684857+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:28.684857+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8400,"log_dups_size":0,"ondisk_log_size":8400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28529067,"num_objects":3483,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8397,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1421}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8314","reported_seq":12355,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:29.381120+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:29.381120+0000","last_peered":"2024-09-15T03:12:29.381120+0000","last_clean":"2024-09-15T03:12:29.381120+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:29.381120+0000","last_undegraded":"2024-09-15T03:12:29.381120+0000","last_fullsized":"2024-09-15T03:12:29.381120+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8314,"log_dups_size":0,"ondisk_log_size":8314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29367097,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8313,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8470","reported_seq":9234,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.950032+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:28.950032+0000","last_peered":"2024-09-15T03:12:28.950032+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:28.950032+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8470,"log_dups_size":0,"ondisk_log_size":8470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30006073,"num_objects":3663,"num_object_clones":0,"num_object_copies":14652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8469,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1964}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8229","reported_seq":8329,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.785969+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:28.785969+0000","last_peered":"2024-09-15T03:12:28.785969+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:28.785969+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8229,"log_dups_size":0,"ondisk_log_size":8229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28284530,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8227,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1440}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8315","reported_seq":8422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:29.017804+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:29.017804+0000","last_peered":"2024-09-15T03:12:29.017804+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:29.017804+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29226610,"num_objects":3568,"num_object_clones":0,"num_object_copies":14272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8313,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1591}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8416","reported_seq":8529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:28.705138+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:28.705138+0000","last_peered":"2024-09-15T03:12:28.705138+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:28.705138+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29431410,"num_objects":3593,"num_object_clones":0,"num_object_copies":14372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8414,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1534}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":466033236,"num_objects":56894,"num_object_clones":0,"num_object_copies":227576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133813,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4537,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":931844096,"data_stored":931844096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133840,"ondisk_log_size":133840,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530264,"kb_used_data":228240,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93841576,"statfs":{"total":96636764160,"available":96093773824,"internally_reserved":0,"allocated":233717760,"data_stored":233542294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,8,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":4.2960000000000003},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.75600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":4.3630000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.88400000000000001},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":4.3360000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533996,"kb_used_data":228324,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93837844,"statfs":{"total":96636764160,"available":96089952256,"internally_reserved":0,"allocated":233803776,"data_stored":233631456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,1,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:28 2024","interfaces":[{"interface":"back","average":{"1min":3.5680000000000001,"5min":2.7109999999999999,"15min":1.385},"min":{"1min":0.19400000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":31.175000000000001,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.23000000000000001},{"interface":"front","average":{"1min":1.7410000000000001,"5min":1.9750000000000001,"15min":1.2050000000000001},"min":{"1min":0.20100000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.606,"5min":19.329000000000001,"15min":19.329000000000001},"last":2.351}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.84799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.376}]}]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526196,"kb_used_data":228316,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93845644,"statfs":{"total":96636764160,"available":96097939456,"internally_reserved":0,"allocated":233795584,"data_stored":233620118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,4,4,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.17399999999999999},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.318}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.38},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":0.20899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.33800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538200,"kb_used_data":227920,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93833640,"statfs":{"total":96636764160,"available":96085647360,"internally_reserved":0,"allocated":233390080,"data_stored":233214614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.314},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.29399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":233046016,"data_stored":233046016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":232808448,"data_stored":232808448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232935424,"data_stored":232935424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233054208,"data_stored":233054208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:37.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:39.532 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:39.557 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:39.588 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-15T03:12:38.848434+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":457734740,"num_objects":55881,"num_object_clones":0,"num_object_copies":223524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134826,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4581,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134853,"ondisk_log_size":134853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2112540,"kb_used_data":896692,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375374820,"statfs":{"total":386547056640,"available":384383815680,"internally_reserved":0,"allocated":918212608,"data_stored":917513890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,4,10,17,10,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16818176,"num_objects":-2053,"num_object_clones":0,"num_object_copies":-8212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011314"},"pg_stats":[{"pgid":"1.f","version":"35'8345","reported_seq":8411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.913041+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:33.913041+0000","last_peered":"2024-09-15T03:12:33.913041+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:33.913041+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8345,"log_dups_size":0,"ondisk_log_size":8345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29204480,"num_objects":3565,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8345,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8155","reported_seq":8249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.757161+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:33.757161+0000","last_peered":"2024-09-15T03:12:33.757161+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:33.757161+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8155,"log_dups_size":0,"ondisk_log_size":8155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26793586,"num_objects":3271,"num_object_clones":0,"num_object_copies":13084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8153,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8667","reported_seq":8775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.944923+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:33.944923+0000","last_peered":"2024-09-15T03:12:33.944923+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:33.944923+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8667,"log_dups_size":0,"ondisk_log_size":8667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29104977,"num_objects":3554,"num_object_clones":0,"num_object_copies":14216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8666,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1463}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8170","reported_seq":12142,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:34.460579+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:34.460579+0000","last_peered":"2024-09-15T03:12:34.460579+0000","last_clean":"2024-09-15T03:12:34.460579+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:34.460579+0000","last_undegraded":"2024-09-15T03:12:34.460579+0000","last_fullsized":"2024-09-15T03:12:34.460579+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28325490,"num_objects":3458,"num_object_clones":0,"num_object_copies":13832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8539","reported_seq":8645,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.771099+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:33.771099+0000","last_peered":"2024-09-15T03:12:33.771099+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:33.771099+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8539,"log_dups_size":0,"ondisk_log_size":8539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29013618,"num_objects":3542,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8537,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8489","reported_seq":8573,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.950721+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:33.950721+0000","last_peered":"2024-09-15T03:12:33.950721+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:33.950721+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8489,"log_dups_size":0,"ondisk_log_size":8489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28745728,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8489,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1471}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8464","reported_seq":8585,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.915782+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:33.915782+0000","last_peered":"2024-09-15T03:12:33.915782+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:33.915782+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8464,"log_dups_size":0,"ondisk_log_size":8464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28620402,"num_objects":3494,"num_object_clones":0,"num_object_copies":13976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8462,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1468}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8550","reported_seq":8667,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.767254+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:33.767254+0000","last_peered":"2024-09-15T03:12:33.767254+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:33.767254+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8550,"log_dups_size":0,"ondisk_log_size":8550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28956274,"num_objects":3535,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8548,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1479}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8470","reported_seq":8561,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.919098+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:33.919098+0000","last_peered":"2024-09-15T03:12:33.919098+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:33.919098+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8470,"log_dups_size":0,"ondisk_log_size":8470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28667108,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8472","reported_seq":8591,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.830375+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:33.830375+0000","last_peered":"2024-09-15T03:12:33.830375+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:33.830375+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8472,"log_dups_size":0,"ondisk_log_size":8472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28636786,"num_objects":3496,"num_object_clones":0,"num_object_copies":13984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8470,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1454}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8467","reported_seq":8560,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.847968+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:33.847968+0000","last_peered":"2024-09-15T03:12:33.847968+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:33.847968+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8467,"log_dups_size":0,"ondisk_log_size":8467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27980203,"num_objects":3416,"num_object_clones":0,"num_object_copies":13664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8464,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1354}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8362","reported_seq":12403,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:34.527645+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:34.527645+0000","last_peered":"2024-09-15T03:12:34.527645+0000","last_clean":"2024-09-15T03:12:34.527645+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:34.527645+0000","last_undegraded":"2024-09-15T03:12:34.527645+0000","last_fullsized":"2024-09-15T03:12:34.527645+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28973881,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8361,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8537","reported_seq":9389,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.959025+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:33.959025+0000","last_peered":"2024-09-15T03:12:33.959025+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:33.959025+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8537,"log_dups_size":0,"ondisk_log_size":8537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29457209,"num_objects":3596,"num_object_clones":0,"num_object_copies":14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1655,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8536,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":386,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1941}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8291","reported_seq":8391,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.688982+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:33.688982+0000","last_peered":"2024-09-15T03:12:33.688982+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:33.688982+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8291,"log_dups_size":0,"ondisk_log_size":8291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27776626,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8289,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1378}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8390","reported_seq":8497,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.984526+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:33.984526+0000","last_peered":"2024-09-15T03:12:33.984526+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:33.984526+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8390,"log_dups_size":0,"ondisk_log_size":8390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28612210,"num_objects":3493,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8388,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1516}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8485","reported_seq":8597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:33.840587+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:33.840587+0000","last_peered":"2024-09-15T03:12:33.840587+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:33.840587+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8485,"log_dups_size":0,"ondisk_log_size":8485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28866162,"num_objects":3524,"num_object_clones":0,"num_object_copies":14096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1465}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":457734740,"num_objects":55881,"num_object_clones":0,"num_object_copies":223524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134826,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4581,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":915136512,"data_stored":915136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134853,"ondisk_log_size":134853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530436,"kb_used_data":224244,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93841404,"statfs":{"total":96636764160,"available":96093597696,"internally_reserved":0,"allocated":229625856,"data_stored":229450390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.39200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.436},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":1.5660000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534156,"kb_used_data":224324,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93837684,"statfs":{"total":96636764160,"available":96089788416,"internally_reserved":0,"allocated":229707776,"data_stored":229535456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,3,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":1.9470000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":1.8169999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":6.8639999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":6.8979999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:11:33 2024","interfaces":[{"interface":"back","average":{"1min":2.1160000000000001,"5min":2.3079999999999998,"15min":1.1719999999999999},"min":{"1min":0.29199999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.079000000000001,"5min":21.600999999999999,"15min":21.600999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.881,"5min":2.3180000000000001,"15min":1.2849999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":10.861000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526240,"kb_used_data":224216,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93845600,"statfs":{"total":96636764160,"available":96097894400,"internally_reserved":0,"allocated":229597184,"data_stored":229421718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,11,2,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.188},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.32400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":1.7250000000000001},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":2.0070000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":3.2559999999999998},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521708,"kb_used_data":223908,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93850132,"statfs":{"total":96636764160,"available":96102535168,"internally_reserved":0,"allocated":229281792,"data_stored":229106326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,1,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.4550000000000001},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.4710000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.39300000000000002},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.82299999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.32600000000000001},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228917248,"data_stored":228917248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228585472,"data_stored":228585472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228790272,"data_stored":228790272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":228843520,"data_stored":228843520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:39.589 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:41.278 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:41.288 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:41.420 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-15T03:12:40.848902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":449337940,"num_objects":54856,"num_object_clones":0,"num_object_copies":219424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135851,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135878,"ondisk_log_size":135878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2103892,"kb_used_data":879716,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375383468,"statfs":{"total":386547056640,"available":384392671232,"internally_reserved":0,"allocated":900829184,"data_stored":900130466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,7,7,21,14,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25214976,"num_objects":-3078,"num_object_clones":0,"num_object_copies":-12312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3078,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008425"},"pg_stats":[{"pgid":"1.f","version":"35'8401","reported_seq":8467,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.874415+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:38.874415+0000","last_peered":"2024-09-15T03:12:38.874415+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:38.874415+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8401,"log_dups_size":0,"ondisk_log_size":8401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28745728,"num_objects":3509,"num_object_clones":0,"num_object_copies":14036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8401,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8212","reported_seq":8306,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.785293+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:38.785293+0000","last_peered":"2024-09-15T03:12:38.785293+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:38.785293+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8212,"log_dups_size":0,"ondisk_log_size":8212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26326642,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8722","reported_seq":8830,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.791321+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:38.791321+0000","last_peered":"2024-09-15T03:12:38.791321+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:38.791321+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8722,"log_dups_size":0,"ondisk_log_size":8722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28654417,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8721,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8236","reported_seq":12208,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:39.469502+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:39.469502+0000","last_peered":"2024-09-15T03:12:39.469502+0000","last_clean":"2024-09-15T03:12:39.469502+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:39.469502+0000","last_undegraded":"2024-09-15T03:12:39.469502+0000","last_fullsized":"2024-09-15T03:12:39.469502+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27784818,"num_objects":3392,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8618","reported_seq":8724,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.821326+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:38.821326+0000","last_peered":"2024-09-15T03:12:38.821326+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:38.821326+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28366450,"num_objects":3463,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8616,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8566","reported_seq":8650,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.651005+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:38.651005+0000","last_peered":"2024-09-15T03:12:38.651005+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:38.651005+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8566,"log_dups_size":0,"ondisk_log_size":8566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28114944,"num_objects":3432,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8566,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8523","reported_seq":8644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.594016+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:38.594016+0000","last_peered":"2024-09-15T03:12:38.594016+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:38.594016+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8523,"log_dups_size":0,"ondisk_log_size":8523,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28137074,"num_objects":3435,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8521,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1409}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8617","reported_seq":8734,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.889446+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:38.889446+0000","last_peered":"2024-09-15T03:12:38.889446+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:38.889446+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8617,"log_dups_size":0,"ondisk_log_size":8617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28407410,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8615,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8529","reported_seq":8620,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.909991+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:38.909991+0000","last_peered":"2024-09-15T03:12:38.909991+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:38.909991+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8529,"log_dups_size":0,"ondisk_log_size":8529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28183780,"num_objects":3441,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8526,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8541","reported_seq":8660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:39.003199+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:39.003199+0000","last_peered":"2024-09-15T03:12:39.003199+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:39.003199+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28071538,"num_objects":3427,"num_object_clones":0,"num_object_copies":13708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8526","reported_seq":8620,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.715743+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:38.715743+0000","last_peered":"2024-09-15T03:12:38.715743+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:38.715743+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8526,"log_dups_size":0,"ondisk_log_size":8526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27496875,"num_objects":3357,"num_object_clones":0,"num_object_copies":13428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8523,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8420","reported_seq":12461,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:39.647533+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:39.647533+0000","last_peered":"2024-09-15T03:12:39.647533+0000","last_clean":"2024-09-15T03:12:39.647533+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:39.647533+0000","last_undegraded":"2024-09-15T03:12:39.647533+0000","last_fullsized":"2024-09-15T03:12:39.647533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8420,"log_dups_size":0,"ondisk_log_size":8420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28498745,"num_objects":3479,"num_object_clones":0,"num_object_copies":13916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8612","reported_seq":9560,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:39.082681+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:39.082681+0000","last_peered":"2024-09-15T03:12:39.082681+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:39.082681+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8612,"log_dups_size":0,"ondisk_log_size":8612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28842809,"num_objects":3521,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8611,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1914}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8351","reported_seq":8451,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.918160+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:38.918160+0000","last_peered":"2024-09-15T03:12:38.918160+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:38.918160+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8351,"log_dups_size":0,"ondisk_log_size":8351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27285106,"num_objects":3331,"num_object_clones":0,"num_object_copies":13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1318}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8456","reported_seq":8563,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:39.083782+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:39.083782+0000","last_peered":"2024-09-15T03:12:39.083782+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:39.083782+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28071538,"num_objects":3427,"num_object_clones":0,"num_object_copies":13708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8454,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8548","reported_seq":8661,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:38.891495+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:38.891495+0000","last_peered":"2024-09-15T03:12:38.891495+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:38.891495+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28350066,"num_objects":3461,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":449337940,"num_objects":54856,"num_object_clones":0,"num_object_copies":219424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54856,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135851,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":898301952,"data_stored":898301952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135878,"ondisk_log_size":135878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530576,"kb_used_data":220232,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93841264,"statfs":{"total":96636764160,"available":96093454336,"internally_reserved":0,"allocated":225517568,"data_stored":225342102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,2,9,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.246}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.36899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.44600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521708,"kb_used_data":220196,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93850132,"statfs":{"total":96636764160,"available":96102535168,"internally_reserved":0,"allocated":225480704,"data_stored":225308384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":9.4909999999999997},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.20699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":4.117},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":4.1799999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":4.1470000000000002},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.29099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529784,"kb_used_data":219424,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93842056,"statfs":{"total":96636764160,"available":96094265344,"internally_reserved":0,"allocated":224690176,"data_stored":224514710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,2,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.315},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.26200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":0.35499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.245}]}]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521824,"kb_used_data":219864,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93850016,"statfs":{"total":96636764160,"available":96102416384,"internally_reserved":0,"allocated":225140736,"data_stored":224965270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":9.8239999999999998},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":10.019}]},{"osd":2,"last update":"Sun Sep 15 03:11:35 2024","interfaces":[{"interface":"back","average":{"1min":3.645,"5min":3.2229999999999999,"15min":1.6970000000000001},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":19.919,"5min":30.802,"15min":30.802},"last":0.92000000000000004},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.0209999999999999,"15min":1.0800000000000001},"min":{"1min":0.254,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":18.012,"5min":18.012,"15min":18.012},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.0070000000000001,"5min":1.9490000000000001,"15min":1.0900000000000001},"min":{"1min":0.255,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":6.9189999999999996,"5min":20.948,"15min":20.948},"last":0.32100000000000001},{"interface":"front","average":{"1min":2.1019999999999999,"5min":1.8180000000000001,"15min":0.91800000000000004},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22700000000000001},"max":{"1min":10.555,"5min":11.6,"15min":11.6},"last":0.42699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224714752,"data_stored":224714752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224354304,"data_stored":224354304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224567296,"data_stored":224567296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224665600,"data_stored":224665600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:41.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:12:41.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8401", 'reported_seq': 8467, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.874415+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:12:38.874415+0000', 'last_peered': '2024-09-15T03:12:38.874415+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:12:38.874415+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8401, 'log_dups_size': 0, 'ondisk_log_size': 8401, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28745728, 'num_objects': 3509, 'num_object_clones': 0, 'num_object_copies': 14036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8401, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1543}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:41.421 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:12:41.421 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8212", 'reported_seq': 8306, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.785293+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:12:38.785293+0000', 'last_peered': '2024-09-15T03:12:38.785293+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:12:38.785293+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8212, 'log_dups_size': 0, 'ondisk_log_size': 8212, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26326642, 'num_objects': 3214, 'num_object_clones': 0, 'num_object_copies': 12856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8210, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8722", 'reported_seq': 8830, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.791321+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:12:38.791321+0000', 'last_peered': '2024-09-15T03:12:38.791321+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:12:38.791321+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8722, 'log_dups_size': 0, 'ondisk_log_size': 8722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28654417, 'num_objects': 3499, 'num_object_clones': 0, 'num_object_copies': 13996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3499, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8721, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1408}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8618", 'reported_seq': 8724, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.821326+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:12:38.821326+0000', 'last_peered': '2024-09-15T03:12:38.821326+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:12:38.821326+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8618, 'log_dups_size': 0, 'ondisk_log_size': 8618, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28366450, 'num_objects': 3463, 'num_object_clones': 0, 'num_object_copies': 13852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8616, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1379}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8566", 'reported_seq': 8650, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.651005+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:12:38.651005+0000', 'last_peered': '2024-09-15T03:12:38.651005+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:12:38.651005+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8566, 'log_dups_size': 0, 'ondisk_log_size': 8566, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28114944, 'num_objects': 3432, 'num_object_clones': 0, 'num_object_copies': 13728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8566, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8523", 'reported_seq': 8644, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.594016+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:12:38.594016+0000', 'last_peered': '2024-09-15T03:12:38.594016+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:12:38.594016+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8523, 'log_dups_size': 0, 'ondisk_log_size': 8523, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28137074, 'num_objects': 3435, 'num_object_clones': 0, 'num_object_copies': 13740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8521, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1409}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:12:41.422 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8617", 'reported_seq': 8734, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.889446+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:12:38.889446+0000', 'last_peered': '2024-09-15T03:12:38.889446+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:12:38.889446+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8617, 'log_dups_size': 0, 'ondisk_log_size': 8617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28407410, 'num_objects': 3468, 'num_object_clones': 0, 'num_object_copies': 13872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8615, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1412}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8529", 'reported_seq': 8620, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.909991+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:12:38.909991+0000', 'last_peered': '2024-09-15T03:12:38.909991+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:12:38.909991+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8529, 'log_dups_size': 0, 'ondisk_log_size': 8529, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28183780, 'num_objects': 3441, 'num_object_clones': 0, 'num_object_copies': 13764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8526, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1414}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8541", 'reported_seq': 8660, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:39.003199+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:12:39.003199+0000', 'last_peered': '2024-09-15T03:12:39.003199+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:12:39.003199+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8541, 'log_dups_size': 0, 'ondisk_log_size': 8541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28071538, 'num_objects': 3427, 'num_object_clones': 0, 'num_object_copies': 13708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8539, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1385}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8526", 'reported_seq': 8620, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.715743+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:12:38.715743+0000', 'last_peered': '2024-09-15T03:12:38.715743+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:12:38.715743+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8526, 'log_dups_size': 0, 'ondisk_log_size': 8526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27496875, 'num_objects': 3357, 'num_object_clones': 0, 'num_object_copies': 13428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3357, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8523, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8612", 'reported_seq': 9560, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:39.082681+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:12:39.082681+0000', 'last_peered': '2024-09-15T03:12:39.082681+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:12:39.082681+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8612, 'log_dups_size': 0, 'ondisk_log_size': 8612, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28842809, 'num_objects': 3521, 'num_object_clones': 0, 'num_object_copies': 14084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1607, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8611, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 434, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1914}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8351", 'reported_seq': 8451, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.918160+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:12:38.918160+0000', 'last_peered': '2024-09-15T03:12:38.918160+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:12:38.918160+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8351, 'log_dups_size': 0, 'ondisk_log_size': 8351, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27285106, 'num_objects': 3331, 'num_object_clones': 0, 'num_object_copies': 13324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8349, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:41.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:12:41.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8456", 'reported_seq': 8563, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:39.083782+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:12:39.083782+0000', 'last_peered': '2024-09-15T03:12:39.083782+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:12:39.083782+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8456, 'log_dups_size': 0, 'ondisk_log_size': 8456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28071538, 'num_objects': 3427, 'num_object_clones': 0, 'num_object_copies': 13708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8454, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:41.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:12:41.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8548", 'reported_seq': 8661, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:38.891495+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:12:38.891495+0000', 'last_peered': '2024-09-15T03:12:38.891495+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:12:38.891495+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8548, 'log_dups_size': 0, 'ondisk_log_size': 8548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28350066, 'num_objects': 3461, 'num_object_clones': 0, 'num_object_copies': 13844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8546, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:41.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:12:43.969 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:43.969 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28055/219424 objects degraded (12.786%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":487,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54856,"data_bytes":449337940,"bytes_used":2154385408,"bytes_avail":384392671232,"bytes_total":386547056640,"degraded_objects":28055,"degraded_total":219424,"degraded_ratio":0.12785748140586262,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":169},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:12:43.995 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:12:43.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:45.719 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:45.722 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:45.762 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-15T03:12:44.849896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":441162324,"num_objects":53858,"num_object_clones":0,"num_object_copies":215432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136849,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4667,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136876,"ondisk_log_size":136876,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2071212,"kb_used_data":863688,"kb_used_omap":14,"kb_used_meta":1207473,"kb_avail":375416148,"statfs":{"total":386547056640,"available":384426135552,"internally_reserved":0,"allocated":884416512,"data_stored":883717794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1236452379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,9,18,9,7,5,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24870912,"num_objects":-3036,"num_object_clones":0,"num_object_copies":-12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3036,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008450"},"pg_stats":[{"pgid":"1.f","version":"35'8479","reported_seq":8545,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:44.030035+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:44.030035+0000","last_peered":"2024-09-15T03:12:44.030035+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:44.030035+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8479,"log_dups_size":0,"ondisk_log_size":8479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28106752,"num_objects":3431,"num_object_clones":0,"num_object_copies":13724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8479,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8258","reported_seq":8352,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.793079+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:43.793079+0000","last_peered":"2024-09-15T03:12:43.793079+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:43.793079+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25949810,"num_objects":3168,"num_object_clones":0,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1184}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8792","reported_seq":8900,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:44.001586+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:44.001586+0000","last_peered":"2024-09-15T03:12:44.001586+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:44.001586+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28080977,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8791,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8301","reported_seq":12273,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:44.477352+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:44.477352+0000","last_peered":"2024-09-15T03:12:44.477352+0000","last_clean":"2024-09-15T03:12:44.477352+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:44.477352+0000","last_undegraded":"2024-09-15T03:12:44.477352+0000","last_fullsized":"2024-09-15T03:12:44.477352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8301,"log_dups_size":0,"ondisk_log_size":8301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27252338,"num_objects":3327,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8677","reported_seq":8783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.676261+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:43.676261+0000","last_peered":"2024-09-15T03:12:43.676261+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:43.676261+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8677,"log_dups_size":0,"ondisk_log_size":8677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27883122,"num_objects":3404,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8675,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8635","reported_seq":8719,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.738628+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:43.738628+0000","last_peered":"2024-09-15T03:12:43.738628+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:43.738628+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8635,"log_dups_size":0,"ondisk_log_size":8635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27549696,"num_objects":3363,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1325}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8590","reported_seq":8711,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.710122+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:43.710122+0000","last_peered":"2024-09-15T03:12:43.710122+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:43.710122+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8590,"log_dups_size":0,"ondisk_log_size":8590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27588210,"num_objects":3368,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8588,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1342}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8678","reported_seq":8795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.854208+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:43.854208+0000","last_peered":"2024-09-15T03:12:43.854208+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:43.854208+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8678,"log_dups_size":0,"ondisk_log_size":8678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27907698,"num_objects":3407,"num_object_clones":0,"num_object_copies":13628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8584","reported_seq":8675,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.679000+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:43.679000+0000","last_peered":"2024-09-15T03:12:43.679000+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:43.679000+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27733220,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8607","reported_seq":8726,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:44.056764+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:44.056764+0000","last_peered":"2024-09-15T03:12:44.056764+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:44.056764+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8607,"log_dups_size":0,"ondisk_log_size":8607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530866,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8605,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8592","reported_seq":8686,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.898684+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:43.898684+0000","last_peered":"2024-09-15T03:12:43.898684+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:43.898684+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8592,"log_dups_size":0,"ondisk_log_size":8592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26956203,"num_objects":3291,"num_object_clones":0,"num_object_copies":13164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8490","reported_seq":12528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:44.444538+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:44.444538+0000","last_peered":"2024-09-15T03:12:44.444538+0000","last_clean":"2024-09-15T03:12:44.444538+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:44.444538+0000","last_undegraded":"2024-09-15T03:12:44.444538+0000","last_fullsized":"2024-09-15T03:12:44.444538+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8490,"log_dups_size":0,"ondisk_log_size":8490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27925305,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8489,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8677","reported_seq":9701,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:44.078358+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:44.078358+0000","last_peered":"2024-09-15T03:12:44.078358+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:44.078358+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8677,"log_dups_size":0,"ondisk_log_size":8677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28310329,"num_objects":3456,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1887}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8406","reported_seq":8506,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.856767+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:43.856767+0000","last_peered":"2024-09-15T03:12:43.856767+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:43.856767+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834546,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8404,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8506","reported_seq":8611,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.914661+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:43.914661+0000","last_peered":"2024-09-15T03:12:43.914661+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:43.914661+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8506,"log_dups_size":0,"ondisk_log_size":8506,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27661938,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8504,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1400}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8604","reported_seq":8717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:43.958320+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:43.958320+0000","last_peered":"2024-09-15T03:12:43.958320+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:43.958320+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8604,"log_dups_size":0,"ondisk_log_size":8604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27891314,"num_objects":3405,"num_object_clones":0,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8602,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":441162324,"num_objects":53858,"num_object_clones":0,"num_object_copies":215432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136849,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4667,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":881991680,"data_stored":881991680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136876,"ondisk_log_size":136876,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514052,"kb_used_data":216188,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93857788,"statfs":{"total":96636764160,"available":96110374912,"internally_reserved":0,"allocated":221376512,"data_stored":221201046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,7,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.8799999999999999,"15min":1.756},"min":{"1min":0.186,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":8.2240000000000002,"5min":17.408999999999999,"15min":17.408999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":2.4590000000000001,"5min":2.089,"15min":2.0739999999999998},"min":{"1min":0.19800000000000001,"5min":0.185,"15min":0.185},"max":{"1min":26.143000000000001,"5min":26.143000000000001,"15min":26.143000000000001},"last":2.165}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.438},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.38500000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:11:41 2024","interfaces":[{"interface":"back","average":{"1min":5.4820000000000002,"5min":3.2559999999999998,"15min":1.4399999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.20200000000000001},"max":{"1min":40.784999999999997,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":3.6859999999999999,"5min":2.3650000000000002,"15min":1.4119999999999999},"min":{"1min":0.22800000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":32.377000000000002,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.30199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521812,"kb_used_data":216140,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93850028,"statfs":{"total":96636764160,"available":96102428672,"internally_reserved":0,"allocated":221327360,"data_stored":221155040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":9.4909999999999997},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.20699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":4.117},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":4.1799999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":4.1470000000000002},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.29099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513272,"kb_used_data":215404,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93858568,"statfs":{"total":96636764160,"available":96111173632,"internally_reserved":0,"allocated":220573696,"data_stored":220398230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,6,1,3,3,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.26000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":3.427},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":0.40000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":3.8650000000000002},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522076,"kb_used_data":215956,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93849764,"statfs":{"total":96636764160,"available":96102158336,"internally_reserved":0,"allocated":221138944,"data_stored":220963478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":3.137},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":3.1619999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":0.52500000000000002},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.373}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.308},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.34000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220618752,"data_stored":220618752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":220258304,"data_stored":220258304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":220520448,"data_stored":220520448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":220594176,"data_stored":220594176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:48.764 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:51.008 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:51.037 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:51.066 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-15T03:12:50.851395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":432609876,"num_objects":52814,"num_object_clones":0,"num_object_copies":211256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137920,"ondisk_log_size":137920,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2067236,"kb_used_data":847236,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375420124,"statfs":{"total":386547056640,"available":384430206976,"internally_reserved":0,"allocated":867569664,"data_stored":866870946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,2,11,11,4,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25124864,"num_objects":-3067,"num_object_clones":0,"num_object_copies":-12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3067,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"1.f","version":"35'8555","reported_seq":8621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.959376+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:48.959376+0000","last_peered":"2024-09-15T03:12:48.959376+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:48.959376+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8555,"log_dups_size":0,"ondisk_log_size":8555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27484160,"num_objects":3355,"num_object_clones":0,"num_object_copies":13420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8309","reported_seq":8403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.603728+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:48.603728+0000","last_peered":"2024-09-15T03:12:48.603728+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:48.603728+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25532018,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8859","reported_seq":8966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.216299+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:49.216299+0000","last_peered":"2024-09-15T03:12:49.216299+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:49.216299+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8859,"log_dups_size":0,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27532113,"num_objects":3362,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3362,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8858,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1271}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8357","reported_seq":12329,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:49.745775+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:49.745775+0000","last_peered":"2024-09-15T03:12:49.745775+0000","last_clean":"2024-09-15T03:12:49.745775+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:49.745775+0000","last_undegraded":"2024-09-15T03:12:49.745775+0000","last_fullsized":"2024-09-15T03:12:49.745775+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26793586,"num_objects":3271,"num_object_clones":0,"num_object_copies":13084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8355,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8742","reported_seq":8848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.928532+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:48.928532+0000","last_peered":"2024-09-15T03:12:48.928532+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:48.928532+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27350642,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1255}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8716","reported_seq":8800,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.052775+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:49.052775+0000","last_peered":"2024-09-15T03:12:49.052775+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:49.052775+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26886144,"num_objects":3282,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8716,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8664","reported_seq":8785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.973124+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:48.973124+0000","last_peered":"2024-09-15T03:12:48.973124+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:48.973124+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26982002,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8753","reported_seq":8870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.968520+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:48.968520+0000","last_peered":"2024-09-15T03:12:48.968520+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:48.968520+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8753,"log_dups_size":0,"ondisk_log_size":8753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27293298,"num_objects":3332,"num_object_clones":0,"num_object_copies":13328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8751,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8644","reported_seq":8733,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.051349+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:49.051349+0000","last_peered":"2024-09-15T03:12:49.051349+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:49.051349+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8644,"log_dups_size":0,"ondisk_log_size":8644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27241700,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8663","reported_seq":8782,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.010882+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:49.010882+0000","last_peered":"2024-09-15T03:12:49.010882+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:49.010882+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27072114,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8659","reported_seq":8752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.073347+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:49.073347+0000","last_peered":"2024-09-15T03:12:49.073347+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:49.073347+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8659,"log_dups_size":0,"ondisk_log_size":8659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26407339,"num_objects":3224,"num_object_clones":0,"num_object_copies":12896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8656,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8556","reported_seq":12596,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:49.850946+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:49.850946+0000","last_peered":"2024-09-15T03:12:49.850946+0000","last_clean":"2024-09-15T03:12:49.850946+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:49.850946+0000","last_undegraded":"2024-09-15T03:12:49.850946+0000","last_fullsized":"2024-09-15T03:12:49.850946+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27384633,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8742","reported_seq":9860,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.215932+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:49.215932+0000","last_peered":"2024-09-15T03:12:49.215932+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:49.215932+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27777849,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1869}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8468","reported_seq":8568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.968730+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:48.968730+0000","last_peered":"2024-09-15T03:12:48.968730+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:48.968730+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8468,"log_dups_size":0,"ondisk_log_size":8468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26326642,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8466,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1201}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8576","reported_seq":8683,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.056344+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:49.056344+0000","last_peered":"2024-09-15T03:12:49.056344+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:49.056344+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27088498,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8574,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1330}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8657","reported_seq":8770,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.970299+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:48.970299+0000","last_peered":"2024-09-15T03:12:48.970299+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:48.970299+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27457138,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8655,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":432609876,"num_objects":52814,"num_object_clones":0,"num_object_copies":211256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":864874496,"data_stored":864874496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137920,"ondisk_log_size":137920,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518204,"kb_used_data":212020,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93853636,"statfs":{"total":96636764160,"available":96106123264,"internally_reserved":0,"allocated":217108480,"data_stored":216933014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":6.4290000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":10.598000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.38900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521876,"kb_used_data":212044,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93849964,"statfs":{"total":96636764160,"available":96102363136,"internally_reserved":0,"allocated":217133056,"data_stored":216960736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,5,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.45100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.38100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.49199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517448,"kb_used_data":211264,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93854392,"statfs":{"total":96636764160,"available":96106897408,"internally_reserved":0,"allocated":216334336,"data_stored":216158870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,5,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.62},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":5.9320000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.432},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":11.052}]}]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509708,"kb_used_data":211908,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93862132,"statfs":{"total":96636764160,"available":96114823168,"internally_reserved":0,"allocated":216993792,"data_stored":216818326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.593},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.714}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":1.7470000000000001},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":1.917}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.26000000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":216379392,"data_stored":216379392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215900160,"data_stored":215900160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216256512,"data_stored":216256512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216338432,"data_stored":216338432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:51.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:52.915 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:52.963 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:52.993 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-15T03:12:52.851925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":432609876,"num_objects":52814,"num_object_clones":0,"num_object_copies":211256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137920,"ondisk_log_size":137920,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2067236,"kb_used_data":847236,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375420124,"statfs":{"total":386547056640,"available":384430206976,"internally_reserved":0,"allocated":867569664,"data_stored":866870946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,2,11,11,4,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16728064,"num_objects":-2042,"num_object_clones":0,"num_object_copies":-8168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2042,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003022"},"pg_stats":[{"pgid":"1.f","version":"35'8555","reported_seq":8621,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.959376+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:48.959376+0000","last_peered":"2024-09-15T03:12:48.959376+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:48.959376+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8555,"log_dups_size":0,"ondisk_log_size":8555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27484160,"num_objects":3355,"num_object_clones":0,"num_object_copies":13420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8309","reported_seq":8403,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.603728+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:48.603728+0000","last_peered":"2024-09-15T03:12:48.603728+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:48.603728+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8309,"log_dups_size":0,"ondisk_log_size":8309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25532018,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1133}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8859","reported_seq":8966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.216299+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:49.216299+0000","last_peered":"2024-09-15T03:12:49.216299+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:49.216299+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8859,"log_dups_size":0,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27532113,"num_objects":3362,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3362,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8858,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1271}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8357","reported_seq":12329,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:49.745775+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:49.745775+0000","last_peered":"2024-09-15T03:12:49.745775+0000","last_clean":"2024-09-15T03:12:49.745775+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:49.745775+0000","last_undegraded":"2024-09-15T03:12:49.745775+0000","last_fullsized":"2024-09-15T03:12:49.745775+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26793586,"num_objects":3271,"num_object_clones":0,"num_object_copies":13084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8355,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8742","reported_seq":8848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.928532+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:48.928532+0000","last_peered":"2024-09-15T03:12:48.928532+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:48.928532+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27350642,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8740,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1255}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8716","reported_seq":8800,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.052775+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:49.052775+0000","last_peered":"2024-09-15T03:12:49.052775+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:49.052775+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26886144,"num_objects":3282,"num_object_clones":0,"num_object_copies":13128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8716,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1244}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8664","reported_seq":8785,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.973124+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:48.973124+0000","last_peered":"2024-09-15T03:12:48.973124+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:48.973124+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26982002,"num_objects":3294,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8753","reported_seq":8870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.968520+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:48.968520+0000","last_peered":"2024-09-15T03:12:48.968520+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:48.968520+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8753,"log_dups_size":0,"ondisk_log_size":8753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27293298,"num_objects":3332,"num_object_clones":0,"num_object_copies":13328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8751,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1276}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8644","reported_seq":8733,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.051349+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:49.051349+0000","last_peered":"2024-09-15T03:12:49.051349+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:49.051349+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8644,"log_dups_size":0,"ondisk_log_size":8644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27241700,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8641,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1299}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8663","reported_seq":8782,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.010882+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:49.010882+0000","last_peered":"2024-09-15T03:12:49.010882+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:49.010882+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27072114,"num_objects":3305,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8659","reported_seq":8752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.073347+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:49.073347+0000","last_peered":"2024-09-15T03:12:49.073347+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:49.073347+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8659,"log_dups_size":0,"ondisk_log_size":8659,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26407339,"num_objects":3224,"num_object_clones":0,"num_object_copies":12896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8656,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1162}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8556","reported_seq":12596,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:49.850946+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:49.850946+0000","last_peered":"2024-09-15T03:12:49.850946+0000","last_clean":"2024-09-15T03:12:49.850946+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:49.850946+0000","last_undegraded":"2024-09-15T03:12:49.850946+0000","last_fullsized":"2024-09-15T03:12:49.850946+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27384633,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8742","reported_seq":9860,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.215932+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:49.215932+0000","last_peered":"2024-09-15T03:12:49.215932+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:49.215932+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8742,"log_dups_size":0,"ondisk_log_size":8742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27777849,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1869}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8468","reported_seq":8568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.968730+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:48.968730+0000","last_peered":"2024-09-15T03:12:48.968730+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:48.968730+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8468,"log_dups_size":0,"ondisk_log_size":8468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26326642,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8466,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1201}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8576","reported_seq":8683,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:49.056344+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:49.056344+0000","last_peered":"2024-09-15T03:12:49.056344+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:49.056344+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27088498,"num_objects":3307,"num_object_clones":0,"num_object_copies":13228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8574,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1330}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8657","reported_seq":8770,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:48.970299+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:48.970299+0000","last_peered":"2024-09-15T03:12:48.970299+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:48.970299+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27457138,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8655,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":432609876,"num_objects":52814,"num_object_clones":0,"num_object_copies":211256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52814,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137893,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4714,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":864874496,"data_stored":864874496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137920,"ondisk_log_size":137920,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518204,"kb_used_data":212020,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93853636,"statfs":{"total":96636764160,"available":96106123264,"internally_reserved":0,"allocated":217108480,"data_stored":216933014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":6.4290000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":10.598000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.38900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521876,"kb_used_data":212044,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93849964,"statfs":{"total":96636764160,"available":96102363136,"internally_reserved":0,"allocated":217133056,"data_stored":216960736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,5,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.45100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.38100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.49199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517448,"kb_used_data":211264,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93854392,"statfs":{"total":96636764160,"available":96106897408,"internally_reserved":0,"allocated":216334336,"data_stored":216158870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,5,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.62},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":5.9320000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.432},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":11.052}]}]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509708,"kb_used_data":211908,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93862132,"statfs":{"total":96636764160,"available":96114823168,"internally_reserved":0,"allocated":216993792,"data_stored":216818326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.593},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.714}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":1.7470000000000001},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":1.917}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.26000000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":216379392,"data_stored":216379392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215900160,"data_stored":215900160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216256512,"data_stored":216256512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":216338432,"data_stored":216338432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:52.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:12:52.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8555", 'reported_seq': 8621, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:48.959376+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:12:48.959376+0000', 'last_peered': '2024-09-15T03:12:48.959376+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:12:48.959376+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8555, 'log_dups_size': 0, 'ondisk_log_size': 8555, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27484160, 'num_objects': 3355, 'num_object_clones': 0, 'num_object_copies': 13420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8555, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1389}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:52.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:12:52.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8309", 'reported_seq': 8403, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:48.603728+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:12:48.603728+0000', 'last_peered': '2024-09-15T03:12:48.603728+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:12:48.603728+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8309, 'log_dups_size': 0, 'ondisk_log_size': 8309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25532018, 'num_objects': 3117, 'num_object_clones': 0, 'num_object_copies': 12468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8307, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1133}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8859", 'reported_seq': 8966, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:49.216299+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:12:49.216299+0000', 'last_peered': '2024-09-15T03:12:49.216299+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:12:49.216299+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8859, 'log_dups_size': 0, 'ondisk_log_size': 8859, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27532113, 'num_objects': 3362, 'num_object_clones': 0, 'num_object_copies': 13448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3362, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8858, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1271}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8742", 'reported_seq': 8848, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:48.928532+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:12:48.928532+0000', 'last_peered': '2024-09-15T03:12:48.928532+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:12:48.928532+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8742, 'log_dups_size': 0, 'ondisk_log_size': 8742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27350642, 'num_objects': 3339, 'num_object_clones': 0, 'num_object_copies': 13356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8740, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1255}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8716", 'reported_seq': 8800, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:49.052775+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:12:49.052775+0000', 'last_peered': '2024-09-15T03:12:49.052775+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:12:49.052775+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8716, 'log_dups_size': 0, 'ondisk_log_size': 8716, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26886144, 'num_objects': 3282, 'num_object_clones': 0, 'num_object_copies': 13128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3282, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8716, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1244}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8664", 'reported_seq': 8785, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:48.973124+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:12:48.973124+0000', 'last_peered': '2024-09-15T03:12:48.973124+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:12:48.973124+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8664, 'log_dups_size': 0, 'ondisk_log_size': 8664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26982002, 'num_objects': 3294, 'num_object_clones': 0, 'num_object_copies': 13176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8662, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1268}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:12:52.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8753", 'reported_seq': 8870, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:48.968520+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:12:48.968520+0000', 'last_peered': '2024-09-15T03:12:48.968520+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:12:48.968520+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8753, 'log_dups_size': 0, 'ondisk_log_size': 8753, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27293298, 'num_objects': 3332, 'num_object_clones': 0, 'num_object_copies': 13328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8751, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1276}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8644", 'reported_seq': 8733, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:49.051349+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:12:49.051349+0000', 'last_peered': '2024-09-15T03:12:49.051349+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:12:49.051349+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8644, 'log_dups_size': 0, 'ondisk_log_size': 8644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27241700, 'num_objects': 3326, 'num_object_clones': 0, 'num_object_copies': 13304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3326, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8641, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1299}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8663", 'reported_seq': 8782, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:49.010882+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:12:49.010882+0000', 'last_peered': '2024-09-15T03:12:49.010882+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:12:49.010882+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8663, 'log_dups_size': 0, 'ondisk_log_size': 8663, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27072114, 'num_objects': 3305, 'num_object_clones': 0, 'num_object_copies': 13220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8661, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1263}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8659", 'reported_seq': 8752, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:49.073347+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:12:49.073347+0000', 'last_peered': '2024-09-15T03:12:49.073347+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:12:49.073347+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8659, 'log_dups_size': 0, 'ondisk_log_size': 8659, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26407339, 'num_objects': 3224, 'num_object_clones': 0, 'num_object_copies': 12896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8656, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1162}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8742", 'reported_seq': 9860, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:49.215932+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:12:49.215932+0000', 'last_peered': '2024-09-15T03:12:49.215932+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:12:49.215932+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8742, 'log_dups_size': 0, 'ondisk_log_size': 8742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27777849, 'num_objects': 3391, 'num_object_clones': 0, 'num_object_copies': 13564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1522, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8741, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 519, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8468", 'reported_seq': 8568, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:48.968730+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:12:48.968730+0000', 'last_peered': '2024-09-15T03:12:48.968730+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:12:48.968730+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8468, 'log_dups_size': 0, 'ondisk_log_size': 8468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26326642, 'num_objects': 3214, 'num_object_clones': 0, 'num_object_copies': 12856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8466, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:12:52.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:12:52.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8576", 'reported_seq': 8683, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:49.056344+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:12:49.056344+0000', 'last_peered': '2024-09-15T03:12:49.056344+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:12:49.056344+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8576, 'log_dups_size': 0, 'ondisk_log_size': 8576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27088498, 'num_objects': 3307, 'num_object_clones': 0, 'num_object_copies': 13228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8574, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1330}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:12:52.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:12:52.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8657", 'reported_seq': 8770, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:48.970299+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:12:48.970299+0000', 'last_peered': '2024-09-15T03:12:48.970299+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:12:48.970299+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8657, 'log_dups_size': 0, 'ondisk_log_size': 8657, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27457138, 'num_objects': 3352, 'num_object_clones': 0, 'num_object_copies': 13408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8655, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:12:52.997 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:12:55.032 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:55.032 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27923/207848 objects degraded (13.434%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":498,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51962,"data_bytes":425630292,"bytes_used":2100301824,"bytes_avail":384446754816,"bytes_total":386547056640,"degraded_objects":27923,"degraded_total":207848,"degraded_ratio":0.13434336630614679,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":132,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":241},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:12:55.122 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:12:55.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:12:56.698 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:12:56.714 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:12:56.808 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-15T03:12:54.852389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":425630292,"num_objects":51962,"num_object_clones":0,"num_object_copies":207848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138745,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4761,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138772,"ondisk_log_size":138772,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2051076,"kb_used_data":835212,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375436284,"statfs":{"total":386547056640,"available":384446754816,"internally_reserved":0,"allocated":855257088,"data_stored":854558370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,10,2,14,15,9,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23707648,"num_objects":-2894,"num_object_clones":0,"num_object_copies":-11576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2894,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002960"},"pg_stats":[{"pgid":"1.f","version":"35'8610","reported_seq":8676,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:53.767161+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:53.767161+0000","last_peered":"2024-09-15T03:12:53.767161+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:53.767161+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8610,"log_dups_size":0,"ondisk_log_size":8610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27033600,"num_objects":3300,"num_object_clones":0,"num_object_copies":13200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8610,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1334}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8376","reported_seq":8470,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:53.893363+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:53.893363+0000","last_peered":"2024-09-15T03:12:53.893363+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:53.893363+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8376,"log_dups_size":0,"ondisk_log_size":8376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24983154,"num_objects":3050,"num_object_clones":0,"num_object_copies":12200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8374,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1066}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8921","reported_seq":9029,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:54.064489+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:54.064489+0000","last_peered":"2024-09-15T03:12:54.064489+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:54.064489+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8921,"log_dups_size":0,"ondisk_log_size":8921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27024209,"num_objects":3300,"num_object_clones":0,"num_object_copies":13200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8920,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1209}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8357","reported_seq":12329,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:49.745775+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:49.745775+0000","last_peered":"2024-09-15T03:12:49.745775+0000","last_clean":"2024-09-15T03:12:49.745775+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:49.745775+0000","last_undegraded":"2024-09-15T03:12:49.745775+0000","last_fullsized":"2024-09-15T03:12:49.745775+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8357,"log_dups_size":0,"ondisk_log_size":8357,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26793586,"num_objects":3271,"num_object_clones":0,"num_object_copies":13084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8355,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8803","reported_seq":8908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:54.004955+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:54.004955+0000","last_peered":"2024-09-15T03:12:54.004955+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:54.004955+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8803,"log_dups_size":0,"ondisk_log_size":8803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26850930,"num_objects":3278,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8801,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8786","reported_seq":8870,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:54.111004+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:54.111004+0000","last_peered":"2024-09-15T03:12:54.111004+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:54.111004+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8786,"log_dups_size":0,"ondisk_log_size":8786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26312704,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8786,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1174}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8727","reported_seq":8847,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:53.884270+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:53.884270+0000","last_peered":"2024-09-15T03:12:53.884270+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:53.884270+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8727,"log_dups_size":0,"ondisk_log_size":8727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26465906,"num_objects":3231,"num_object_clones":0,"num_object_copies":12924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8725,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1205}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8810","reported_seq":8927,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:53.936300+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:53.936300+0000","last_peered":"2024-09-15T03:12:53.936300+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:53.936300+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8810,"log_dups_size":0,"ondisk_log_size":8810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26826354,"num_objects":3275,"num_object_clones":0,"num_object_copies":13100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8808,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8698","reported_seq":8789,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:54.035413+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:54.035413+0000","last_peered":"2024-09-15T03:12:54.035413+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:54.035413+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8698,"log_dups_size":0,"ondisk_log_size":8698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26799332,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8695,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8723","reported_seq":8842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:53.781029+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:53.781029+0000","last_peered":"2024-09-15T03:12:53.781029+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:53.781029+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8723,"log_dups_size":0,"ondisk_log_size":8723,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26580594,"num_objects":3245,"num_object_clones":0,"num_object_copies":12980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1203}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8710","reported_seq":8804,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:53.741387+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:53.741387+0000","last_peered":"2024-09-15T03:12:53.741387+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:53.741387+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25989547,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8556","reported_seq":12596,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:49.850946+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:49.850946+0000","last_peered":"2024-09-15T03:12:49.850946+0000","last_clean":"2024-09-15T03:12:49.850946+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:49.850946+0000","last_undegraded":"2024-09-15T03:12:49.850946+0000","last_fullsized":"2024-09-15T03:12:49.850946+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27384633,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8555,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8802","reported_seq":10014,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:54.226055+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:54.226055+0000","last_peered":"2024-09-15T03:12:54.226055+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:54.226055+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27286329,"num_objects":3331,"num_object_clones":0,"num_object_copies":13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8801,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":566,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1856}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8530","reported_seq":8630,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:54.078807+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:54.078807+0000","last_peered":"2024-09-15T03:12:54.078807+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:54.078807+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25818738,"num_objects":3152,"num_object_clones":0,"num_object_copies":12608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1139}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8642","reported_seq":8749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:54.127787+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:54.127787+0000","last_peered":"2024-09-15T03:12:54.127787+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:54.127787+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8642,"log_dups_size":0,"ondisk_log_size":8642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26547826,"num_objects":3241,"num_object_clones":0,"num_object_copies":12964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8640,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1264}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8721","reported_seq":8834,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:54.008381+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:54.008381+0000","last_peered":"2024-09-15T03:12:54.008381+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:54.008381+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26932850,"num_objects":3288,"num_object_clones":0,"num_object_copies":13152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8719,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1229}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":425630292,"num_objects":51962,"num_object_clones":0,"num_object_copies":207848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138745,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4761,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":852647936,"data_stored":852647936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138772,"ondisk_log_size":138772,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518328,"kb_used_data":207984,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93853512,"statfs":{"total":96636764160,"available":96105996288,"internally_reserved":0,"allocated":212975616,"data_stored":212800150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,7,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.27000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":3.0590000000000002},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.83799999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":11.801},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.878}]}]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505364,"kb_used_data":208012,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93866476,"statfs":{"total":96636764160,"available":96119271424,"internally_reserved":0,"allocated":213004288,"data_stored":212831968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":6.282}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.314},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.371}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":6.3330000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517448,"kb_used_data":211264,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93854392,"statfs":{"total":96636764160,"available":96106897408,"internally_reserved":0,"allocated":216334336,"data_stored":216158870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,5,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.29699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.62},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":5.9320000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.432},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":11.052}]}]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509936,"kb_used_data":207952,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93861904,"statfs":{"total":96636764160,"available":96114589696,"internally_reserved":0,"allocated":212942848,"data_stored":212767382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,1,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.42899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":0.46500000000000002},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":2.4790000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212299776,"data_stored":212299776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215900160,"data_stored":215900160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":212176896,"data_stored":212176896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212271104,"data_stored":212271104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:12:59.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:01.561 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:01.721 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-15T03:13:00.856462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416373332,"num_objects":50832,"num_object_clones":0,"num_object_copies":203328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139875,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4840,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139902,"ondisk_log_size":139902,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2018744,"kb_used_data":815376,"kb_used_omap":14,"kb_used_meta":1203313,"kb_avail":375468616,"statfs":{"total":386547056640,"available":384479862784,"internally_reserved":0,"allocated":834945024,"data_stored":834246306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232192539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,10,2,8,10,8,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24788992,"num_objects":-3026,"num_object_clones":0,"num_object_copies":-12104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005537"},"pg_stats":[{"pgid":"1.f","version":"35'8676","reported_seq":8742,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.991816+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:58.991816+0000","last_peered":"2024-09-15T03:12:58.991816+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:58.991816+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26492928,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8435","reported_seq":8529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.939611+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:58.939611+0000","last_peered":"2024-09-15T03:12:58.939611+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:58.939611+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8435,"log_dups_size":0,"ondisk_log_size":8435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24499826,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8433,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8984","reported_seq":9092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.187244+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:59.187244+0000","last_peered":"2024-09-15T03:12:59.187244+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:59.187244+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8984,"log_dups_size":0,"ondisk_log_size":8984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26508113,"num_objects":3237,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8983,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8453","reported_seq":12425,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:59.969330+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:59.969330+0000","last_peered":"2024-09-15T03:12:59.969330+0000","last_clean":"2024-09-15T03:12:59.969330+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:59.969330+0000","last_undegraded":"2024-09-15T03:12:59.969330+0000","last_fullsized":"2024-09-15T03:12:59.969330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26007154,"num_objects":3175,"num_object_clones":0,"num_object_copies":12700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8865","reported_seq":8971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.694811+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:58.694811+0000","last_peered":"2024-09-15T03:12:58.694811+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:58.694811+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26343026,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8838","reported_seq":8922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.134502+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:59.134502+0000","last_peered":"2024-09-15T03:12:59.134502+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:59.134502+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8838,"log_dups_size":0,"ondisk_log_size":8838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25886720,"num_objects":3160,"num_object_clones":0,"num_object_copies":12640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8801","reported_seq":8922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.068795+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:59.068795+0000","last_peered":"2024-09-15T03:12:59.068795+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:59.068795+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8801,"log_dups_size":0,"ondisk_log_size":8801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25859698,"num_objects":3157,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8799,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8878","reported_seq":8995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.806703+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:58.806703+0000","last_peered":"2024-09-15T03:12:58.806703+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:58.806703+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8878,"log_dups_size":0,"ondisk_log_size":8878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26269298,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8876,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1151}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8768","reported_seq":8859,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.142824+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:59.142824+0000","last_peered":"2024-09-15T03:12:59.142824+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:59.142824+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8768,"log_dups_size":0,"ondisk_log_size":8768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26225892,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8765,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8780","reported_seq":8897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.293442+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:59.293442+0000","last_peered":"2024-09-15T03:12:59.293442+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:59.293442+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26113650,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8778,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8785","reported_seq":8879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.072066+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:59.072066+0000","last_peered":"2024-09-15T03:12:59.072066+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:59.072066+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25375147,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8689","reported_seq":12730,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:59.784630+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:59.784630+0000","last_peered":"2024-09-15T03:12:59.784630+0000","last_clean":"2024-09-15T03:12:59.784630+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:59.784630+0000","last_undegraded":"2024-09-15T03:12:59.784630+0000","last_fullsized":"2024-09-15T03:12:59.784630+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26295097,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8688,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8859","reported_seq":10228,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.307451+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:59.307451+0000","last_peered":"2024-09-15T03:12:59.307451+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:59.307451+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8859,"log_dups_size":0,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26819385,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":645,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8591","reported_seq":8691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.909462+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:58.909462+0000","last_peered":"2024-09-15T03:12:58.909462+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:58.909462+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25319026,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1078}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8710","reported_seq":8817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.034869+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:59.034869+0000","last_peered":"2024-09-15T03:12:59.034869+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:59.034869+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25990770,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8708,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1196}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8790","reported_seq":8903,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.034287+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:59.034287+0000","last_peered":"2024-09-15T03:12:59.034287+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:59.034287+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8790,"log_dups_size":0,"ondisk_log_size":8790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26367602,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8788,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1160}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":416373332,"num_objects":50832,"num_object_clones":0,"num_object_copies":203328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139875,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4840,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":832380928,"data_stored":832380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139902,"ondisk_log_size":139902,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501944,"kb_used_data":204076,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93869896,"statfs":{"total":96636764160,"available":96122773504,"internally_reserved":0,"allocated":208973824,"data_stored":208798358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.216}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":1.268},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.94199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505588,"kb_used_data":204076,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93866252,"statfs":{"total":96636764160,"available":96119042048,"internally_reserved":0,"allocated":208973824,"data_stored":208801504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,1,0,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":2.661}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.50600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":6.75},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":6.7930000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501092,"kb_used_data":203228,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93870748,"statfs":{"total":96636764160,"available":96123645952,"internally_reserved":0,"allocated":208105472,"data_stored":207930006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":3.4769999999999999},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.311}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":3.536}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.32500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510120,"kb_used_data":203996,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93861720,"statfs":{"total":96636764160,"available":96114401280,"internally_reserved":0,"allocated":208891904,"data_stored":208716438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,1,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.42899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":0.46500000000000002},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":2.4790000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208293888,"data_stored":208293888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":207736832,"data_stored":207736832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":208125952,"data_stored":208125952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":208224256,"data_stored":208224256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:01.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:03.200 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:03.247 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:03.257 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-15T03:13:02.856991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416373332,"num_objects":50832,"num_object_clones":0,"num_object_copies":203328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139875,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4840,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139902,"ondisk_log_size":139902,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2018744,"kb_used_data":815376,"kb_used_omap":14,"kb_used_meta":1203313,"kb_avail":375468616,"statfs":{"total":386547056640,"available":384479862784,"internally_reserved":0,"allocated":834945024,"data_stored":834246306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1232192539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,10,2,8,10,8,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16236544,"num_objects":-1982,"num_object_clones":0,"num_object_copies":-7928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1982,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005595"},"pg_stats":[{"pgid":"1.f","version":"35'8676","reported_seq":8742,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.991816+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:12:58.991816+0000","last_peered":"2024-09-15T03:12:58.991816+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:12:58.991816+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8676,"log_dups_size":0,"ondisk_log_size":8676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26492928,"num_objects":3234,"num_object_clones":0,"num_object_copies":12936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1268}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8435","reported_seq":8529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.939611+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:12:58.939611+0000","last_peered":"2024-09-15T03:12:58.939611+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:12:58.939611+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8435,"log_dups_size":0,"ondisk_log_size":8435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24499826,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8433,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'8984","reported_seq":9092,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.187244+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:12:59.187244+0000","last_peered":"2024-09-15T03:12:59.187244+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:12:59.187244+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8984,"log_dups_size":0,"ondisk_log_size":8984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26508113,"num_objects":3237,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8983,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8453","reported_seq":12425,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:59.969330+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:59.969330+0000","last_peered":"2024-09-15T03:12:59.969330+0000","last_clean":"2024-09-15T03:12:59.969330+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:59.969330+0000","last_undegraded":"2024-09-15T03:12:59.969330+0000","last_fullsized":"2024-09-15T03:12:59.969330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26007154,"num_objects":3175,"num_object_clones":0,"num_object_copies":12700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8865","reported_seq":8971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.694811+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:12:58.694811+0000","last_peered":"2024-09-15T03:12:58.694811+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:12:58.694811+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26343026,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1132}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8838","reported_seq":8922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.134502+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:12:59.134502+0000","last_peered":"2024-09-15T03:12:59.134502+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:12:59.134502+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8838,"log_dups_size":0,"ondisk_log_size":8838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25886720,"num_objects":3160,"num_object_clones":0,"num_object_copies":12640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8838,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8801","reported_seq":8922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.068795+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:12:59.068795+0000","last_peered":"2024-09-15T03:12:59.068795+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:12:59.068795+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8801,"log_dups_size":0,"ondisk_log_size":8801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25859698,"num_objects":3157,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8799,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8878","reported_seq":8995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.806703+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:12:58.806703+0000","last_peered":"2024-09-15T03:12:58.806703+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:12:58.806703+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8878,"log_dups_size":0,"ondisk_log_size":8878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26269298,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8876,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1151}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8768","reported_seq":8859,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.142824+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:12:59.142824+0000","last_peered":"2024-09-15T03:12:59.142824+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:12:59.142824+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8768,"log_dups_size":0,"ondisk_log_size":8768,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26225892,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8765,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8780","reported_seq":8897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.293442+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:12:59.293442+0000","last_peered":"2024-09-15T03:12:59.293442+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:12:59.293442+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26113650,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8778,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8785","reported_seq":8879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.072066+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:12:59.072066+0000","last_peered":"2024-09-15T03:12:59.072066+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:12:59.072066+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8785,"log_dups_size":0,"ondisk_log_size":8785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25375147,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8689","reported_seq":12730,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:59.784630+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:59.784630+0000","last_peered":"2024-09-15T03:12:59.784630+0000","last_clean":"2024-09-15T03:12:59.784630+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:59.784630+0000","last_undegraded":"2024-09-15T03:12:59.784630+0000","last_fullsized":"2024-09-15T03:12:59.784630+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26295097,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8688,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8859","reported_seq":10228,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.307451+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:12:59.307451+0000","last_peered":"2024-09-15T03:12:59.307451+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:12:59.307451+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8859,"log_dups_size":0,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26819385,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1396,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":645,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8591","reported_seq":8691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:58.909462+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:12:58.909462+0000","last_peered":"2024-09-15T03:12:58.909462+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:12:58.909462+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25319026,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1078}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8710","reported_seq":8817,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.034869+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:12:59.034869+0000","last_peered":"2024-09-15T03:12:59.034869+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:12:59.034869+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25990770,"num_objects":3173,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8708,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1196}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8790","reported_seq":8903,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:12:59.034287+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:12:59.034287+0000","last_peered":"2024-09-15T03:12:59.034287+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:12:59.034287+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8790,"log_dups_size":0,"ondisk_log_size":8790,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26367602,"num_objects":3219,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8788,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1160}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":416373332,"num_objects":50832,"num_object_clones":0,"num_object_copies":203328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139875,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4840,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":832380928,"data_stored":832380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139902,"ondisk_log_size":139902,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501944,"kb_used_data":204076,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93869896,"statfs":{"total":96636764160,"available":96122773504,"internally_reserved":0,"allocated":208973824,"data_stored":208798358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.216}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":1.268},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.94199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505588,"kb_used_data":204076,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93866252,"statfs":{"total":96636764160,"available":96119042048,"internally_reserved":0,"allocated":208973824,"data_stored":208801504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,1,1,0,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":2.661}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.50600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":6.75},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":6.7930000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501092,"kb_used_data":203228,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93870748,"statfs":{"total":96636764160,"available":96123645952,"internally_reserved":0,"allocated":208105472,"data_stored":207930006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":3.4769999999999999},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.311}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":3.536}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.32500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510120,"kb_used_data":203996,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93861720,"statfs":{"total":96636764160,"available":96114401280,"internally_reserved":0,"allocated":208891904,"data_stored":208716438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,1,3,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.42899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":0.46500000000000002},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.505}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":2.4790000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208293888,"data_stored":208293888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":207736832,"data_stored":207736832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":208125952,"data_stored":208125952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":208224256,"data_stored":208224256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8676", 'reported_seq': 8742, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:58.991816+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:12:58.991816+0000', 'last_peered': '2024-09-15T03:12:58.991816+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:12:58.991816+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8676, 'log_dups_size': 0, 'ondisk_log_size': 8676, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26492928, 'num_objects': 3234, 'num_object_clones': 0, 'num_object_copies': 12936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8676, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1268}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8435", 'reported_seq': 8529, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:58.939611+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:12:58.939611+0000', 'last_peered': '2024-09-15T03:12:58.939611+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:12:58.939611+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8435, 'log_dups_size': 0, 'ondisk_log_size': 8435, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24499826, 'num_objects': 2991, 'num_object_clones': 0, 'num_object_copies': 11964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8433, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'8984", 'reported_seq': 9092, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.187244+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:12:59.187244+0000', 'last_peered': '2024-09-15T03:12:59.187244+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:12:59.187244+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8984, 'log_dups_size': 0, 'ondisk_log_size': 8984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26508113, 'num_objects': 3237, 'num_object_clones': 0, 'num_object_copies': 12948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3237, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8983, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1146}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8865", 'reported_seq': 8971, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:58.694811+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:12:58.694811+0000', 'last_peered': '2024-09-15T03:12:58.694811+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:12:58.694811+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8865, 'log_dups_size': 0, 'ondisk_log_size': 8865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26343026, 'num_objects': 3216, 'num_object_clones': 0, 'num_object_copies': 12864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8863, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1132}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:03.259 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8838", 'reported_seq': 8922, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.134502+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:12:59.134502+0000', 'last_peered': '2024-09-15T03:12:59.134502+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:12:59.134502+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8838, 'log_dups_size': 0, 'ondisk_log_size': 8838, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25886720, 'num_objects': 3160, 'num_object_clones': 0, 'num_object_copies': 12640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8838, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1122}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8801", 'reported_seq': 8922, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.068795+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:12:59.068795+0000', 'last_peered': '2024-09-15T03:12:59.068795+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:12:59.068795+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8801, 'log_dups_size': 0, 'ondisk_log_size': 8801, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25859698, 'num_objects': 3157, 'num_object_clones': 0, 'num_object_copies': 12628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8799, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1131}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'8878", 'reported_seq': 8995, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:58.806703+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:12:58.806703+0000', 'last_peered': '2024-09-15T03:12:58.806703+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:12:58.806703+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8878, 'log_dups_size': 0, 'ondisk_log_size': 8878, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26269298, 'num_objects': 3207, 'num_object_clones': 0, 'num_object_copies': 12828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8876, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1151}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8768", 'reported_seq': 8859, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.142824+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:12:59.142824+0000', 'last_peered': '2024-09-15T03:12:59.142824+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:12:59.142824+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8768, 'log_dups_size': 0, 'ondisk_log_size': 8768, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26225892, 'num_objects': 3202, 'num_object_clones': 0, 'num_object_copies': 12808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8765, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1175}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8780", 'reported_seq': 8897, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.293442+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:12:59.293442+0000', 'last_peered': '2024-09-15T03:12:59.293442+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:12:59.293442+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8780, 'log_dups_size': 0, 'ondisk_log_size': 8780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26113650, 'num_objects': 3188, 'num_object_clones': 0, 'num_object_copies': 12752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8778, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1146}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:13:03.260 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8785", 'reported_seq': 8879, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.072066+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:12:59.072066+0000', 'last_peered': '2024-09-15T03:12:59.072066+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:12:59.072066+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8785, 'log_dups_size': 0, 'ondisk_log_size': 8785, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25375147, 'num_objects': 3098, 'num_object_clones': 0, 'num_object_copies': 12392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8782, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1036}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:03.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:13:03.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8859", 'reported_seq': 10228, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.307451+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:12:59.307451+0000', 'last_peered': '2024-09-15T03:12:59.307451+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:12:59.307451+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8859, 'log_dups_size': 0, 'ondisk_log_size': 8859, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26819385, 'num_objects': 3274, 'num_object_clones': 0, 'num_object_copies': 13096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1396, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8858, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 645, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1878}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:03.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:13:03.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8591", 'reported_seq': 8691, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:58.909462+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:12:58.909462+0000', 'last_peered': '2024-09-15T03:12:58.909462+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:12:58.909462+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8591, 'log_dups_size': 0, 'ondisk_log_size': 8591, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25319026, 'num_objects': 3091, 'num_object_clones': 0, 'num_object_copies': 12364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8589, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1078}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:03.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:13:03.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8710", 'reported_seq': 8817, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.034869+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:12:59.034869+0000', 'last_peered': '2024-09-15T03:12:59.034869+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:12:59.034869+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8710, 'log_dups_size': 0, 'ondisk_log_size': 8710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25990770, 'num_objects': 3173, 'num_object_clones': 0, 'num_object_copies': 12692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8708, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:03.261 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:13:03.261 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8790", 'reported_seq': 8903, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:12:59.034287+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:12:59.034287+0000', 'last_peered': '2024-09-15T03:12:59.034287+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:12:59.034287+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8790, 'log_dups_size': 0, 'ondisk_log_size': 8790, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26367602, 'num_objects': 3219, 'num_object_clones': 0, 'num_object_copies': 12876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8788, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1160}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:03.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:13:05.032 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:05.032 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27786/199872 objects degraded (13.902%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":508,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49968,"data_bytes":409295444,"bytes_used":2072231936,"bytes_avail":384474824704,"bytes_total":386547056640,"degraded_objects":27786,"degraded_total":199872,"degraded_ratio":0.13901897214217099,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":237},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:13:05.093 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:13:05.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:06.719 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:06.726 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:06.748 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-15T03:13:04.857470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":409295444,"num_objects":49968,"num_object_clones":0,"num_object_copies":199872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140739,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4898,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140766,"ondisk_log_size":140766,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2023664,"kb_used_data":803664,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375463696,"statfs":{"total":386547056640,"available":384474824704,"internally_reserved":0,"allocated":822951936,"data_stored":822253218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,11,20,13,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23314432,"num_objects":-2846,"num_object_clones":0,"num_object_copies":-11384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2846,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005545"},"pg_stats":[{"pgid":"1.f","version":"35'8739","reported_seq":8805,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.114018+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:04.114018+0000","last_peered":"2024-09-15T03:13:04.114018+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:04.114018+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8739,"log_dups_size":0,"ondisk_log_size":8739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25976832,"num_objects":3171,"num_object_clones":0,"num_object_copies":12684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8739,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1205}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8492","reported_seq":8586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:03.828803+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:03.828803+0000","last_peered":"2024-09-15T03:13:03.828803+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:03.828803+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8492,"log_dups_size":0,"ondisk_log_size":8492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24032882,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":950}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9064","reported_seq":9172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.263088+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:04.263088+0000","last_peered":"2024-09-15T03:13:04.263088+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:04.263088+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9064,"log_dups_size":0,"ondisk_log_size":9064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25852753,"num_objects":3157,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9063,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1066}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8453","reported_seq":12425,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:59.969330+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:12:59.969330+0000","last_peered":"2024-09-15T03:12:59.969330+0000","last_clean":"2024-09-15T03:12:59.969330+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:12:59.969330+0000","last_undegraded":"2024-09-15T03:12:59.969330+0000","last_fullsized":"2024-09-15T03:12:59.969330+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8453,"log_dups_size":0,"ondisk_log_size":8453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26007154,"num_objects":3175,"num_object_clones":0,"num_object_copies":12700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8926","reported_seq":9032,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.068660+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:04.068660+0000","last_peered":"2024-09-15T03:13:04.068660+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:04.068660+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8926,"log_dups_size":0,"ondisk_log_size":8926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25843314,"num_objects":3155,"num_object_clones":0,"num_object_copies":12620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8924,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1071}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8894","reported_seq":8978,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.005649+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:04.005649+0000","last_peered":"2024-09-15T03:13:04.005649+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:04.005649+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1066}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8859","reported_seq":8980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.029189+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:04.029189+0000","last_peered":"2024-09-15T03:13:04.029189+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:04.029189+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8859,"log_dups_size":0,"ondisk_log_size":8859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25384562,"num_objects":3099,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8857,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1073}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'8954","reported_seq":9069,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.104047+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:04.104047+0000","last_peered":"2024-09-15T03:13:04.104047+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:04.104047+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8954,"log_dups_size":0,"ondisk_log_size":8954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25646706,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8952,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8832","reported_seq":8923,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.069574+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:04.069574+0000","last_peered":"2024-09-15T03:13:04.069574+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:04.069574+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8832,"log_dups_size":0,"ondisk_log_size":8832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25701604,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8829,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8837","reported_seq":8956,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.213501+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:04.213501+0000","last_peered":"2024-09-15T03:13:04.213501+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:04.213501+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8837,"log_dups_size":0,"ondisk_log_size":8837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25646706,"num_objects":3131,"num_object_clones":0,"num_object_copies":12524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8835,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8839","reported_seq":8933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:03.918114+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:03.918114+0000","last_peered":"2024-09-15T03:13:03.918114+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:03.918114+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8839,"log_dups_size":0,"ondisk_log_size":8839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24932779,"num_objects":3044,"num_object_clones":0,"num_object_copies":12176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8836,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":982}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8689","reported_seq":12730,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:12:59.784630+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:12:59.784630+0000","last_peered":"2024-09-15T03:12:59.784630+0000","last_clean":"2024-09-15T03:12:59.784630+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:12:59.784630+0000","last_undegraded":"2024-09-15T03:12:59.784630+0000","last_fullsized":"2024-09-15T03:12:59.784630+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26295097,"num_objects":3210,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8688,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8907","reported_seq":10392,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.341187+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:04.341187+0000","last_peered":"2024-09-15T03:13:04.341187+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:04.341187+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26426169,"num_objects":3226,"num_object_clones":0,"num_object_copies":12904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8906,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1888}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8649","reported_seq":8749,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.069651+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:04.069651+0000","last_peered":"2024-09-15T03:13:04.069651+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:04.069651+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24843890,"num_objects":3033,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1020}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8780","reported_seq":8887,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.032633+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:04.032633+0000","last_peered":"2024-09-15T03:13:04.032633+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:04.032633+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25417330,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8778,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8852","reported_seq":8964,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:04.049800+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:04.049800+0000","last_peered":"2024-09-15T03:13:04.049800+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:04.049800+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25859698,"num_objects":3157,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8850,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1098}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":409295444,"num_objects":49968,"num_object_clones":0,"num_object_copies":199872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140739,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4898,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":820318208,"data_stored":820318208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140766,"ondisk_log_size":140766,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502096,"kb_used_data":200072,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93869744,"statfs":{"total":96636764160,"available":96122617856,"internally_reserved":0,"allocated":204873728,"data_stored":204698262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.253},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.30499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":14.025},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.434}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":13.489000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.28799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510040,"kb_used_data":200208,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93861800,"statfs":{"total":96636764160,"available":96114483200,"internally_reserved":0,"allocated":205012992,"data_stored":204840672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":3.6600000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.249}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":32.295999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":32.340000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.28199999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501092,"kb_used_data":203228,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93870748,"statfs":{"total":96636764160,"available":96123645952,"internally_reserved":0,"allocated":208105472,"data_stored":207930006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.4390000000000001,"5min":1.1899999999999999,"15min":1.9079999999999999},"min":{"1min":0.217,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":10.343,"5min":11.047000000000001,"15min":18.091000000000001},"last":3.4769999999999999},{"interface":"front","average":{"1min":3.2000000000000002,"5min":2.1949999999999998,"15min":1.696},"min":{"1min":0.191,"5min":0.187,"15min":0.187},"max":{"1min":29.294,"5min":29.294,"15min":29.294},"last":0.311}]},{"osd":2,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.9299999999999999,"15min":1.5800000000000001},"min":{"1min":0.25800000000000001,"5min":0.219,"15min":0.219},"max":{"1min":8.0700000000000003,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":3.9900000000000002,"5min":2.4169999999999998,"15min":1.966},"min":{"1min":0.20200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":16.690000000000001,"5min":20.055,"15min":20.055},"last":3.536}]},{"osd":3,"last update":"Sun Sep 15 03:12:09 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.8720000000000001,"15min":1.73},"min":{"1min":0.24199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":10.547000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":2.7040000000000002,"5min":1.867,"15min":1.4370000000000001},"min":{"1min":0.26200000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.8480000000000008,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.32500000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510436,"kb_used_data":200156,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93861404,"statfs":{"total":96636764160,"available":96114077696,"internally_reserved":0,"allocated":204959744,"data_stored":204784278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,8,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":29.763000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":0.38100000000000001},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":4.8810000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.32500000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204288000,"data_stored":204288000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":207736832,"data_stored":207736832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204087296,"data_stored":204087296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204206080,"data_stored":204206080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:09.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:11.717 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:11.745 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:11.768 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-15T03:13:10.858975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":400095828,"num_objects":48845,"num_object_clones":0,"num_object_copies":195380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141862,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4949,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141889,"ondisk_log_size":141889,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1999752,"kb_used_data":783892,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375487608,"statfs":{"total":386547056640,"available":384499310592,"internally_reserved":0,"allocated":802705408,"data_stored":802006690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,10,6,16,8,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24502272,"num_objects":-2991,"num_object_clones":0,"num_object_copies":-11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2991,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005550"},"pg_stats":[{"pgid":"1.f","version":"35'8816","reported_seq":8882,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.110665+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:09.110665+0000","last_peered":"2024-09-15T03:13:09.110665+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:09.110665+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25346048,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8548","reported_seq":8641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.014759+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:09.014759+0000","last_peered":"2024-09-15T03:13:09.014759+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:09.014759+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23574130,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9116","reported_seq":9224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.279146+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:09.279146+0000","last_peered":"2024-09-15T03:13:09.279146+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:09.279146+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9116,"log_dups_size":0,"ondisk_log_size":9116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25426769,"num_objects":3105,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9115,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8581","reported_seq":12552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:10.192484+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:10.192484+0000","last_peered":"2024-09-15T03:13:10.192484+0000","last_clean":"2024-09-15T03:13:10.192484+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:10.192484+0000","last_undegraded":"2024-09-15T03:13:10.192484+0000","last_fullsized":"2024-09-15T03:13:10.192484+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24958578,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8579,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8993","reported_seq":9099,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.114619+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:09.114619+0000","last_peered":"2024-09-15T03:13:09.114619+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:09.114619+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8993,"log_dups_size":0,"ondisk_log_size":8993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25294450,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8949","reported_seq":9032,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.159454+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:09.159454+0000","last_peered":"2024-09-15T03:13:09.159454+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:09.159454+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8949,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8909","reported_seq":9030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:08.996706+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:08.996706+0000","last_peered":"2024-09-15T03:13:08.996706+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:08.996706+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24974962,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9010","reported_seq":9127,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.047536+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:09.047536+0000","last_peered":"2024-09-15T03:13:09.047536+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:09.047536+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9010,"log_dups_size":0,"ondisk_log_size":9010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25187954,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8888","reported_seq":8979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.059779+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:09.059779+0000","last_peered":"2024-09-15T03:13:09.059779+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:09.059779+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8888,"log_dups_size":0,"ondisk_log_size":8888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25242852,"num_objects":3082,"num_object_clones":0,"num_object_copies":12328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8907","reported_seq":9026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:08.900695+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:08.900695+0000","last_peered":"2024-09-15T03:13:08.900695+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:08.900695+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25073266,"num_objects":3061,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8905,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1019}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8914","reported_seq":9007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.118505+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:09.118505+0000","last_peered":"2024-09-15T03:13:09.118505+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:09.118505+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24318379,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8829","reported_seq":12870,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:10.315938+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:10.315938+0000","last_peered":"2024-09-15T03:13:10.315938+0000","last_clean":"2024-09-15T03:13:10.315938+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:10.315938+0000","last_undegraded":"2024-09-15T03:13:10.315938+0000","last_fullsized":"2024-09-15T03:13:10.315938+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25148217,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8976","reported_seq":10564,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.363471+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:09.363471+0000","last_peered":"2024-09-15T03:13:09.363471+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:09.363471+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25860921,"num_objects":3157,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8706","reported_seq":8806,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:08.851061+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:08.851061+0000","last_peered":"2024-09-15T03:13:08.851061+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:08.851061+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8706,"log_dups_size":0,"ondisk_log_size":8706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24376946,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8704,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8829","reported_seq":8936,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.053968+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:09.053968+0000","last_peered":"2024-09-15T03:13:09.053968+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:09.053968+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25015922,"num_objects":3054,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1077}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8918","reported_seq":9031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.044818+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:09.044818+0000","last_peered":"2024-09-15T03:13:09.044818+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:09.044818+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8918,"log_dups_size":0,"ondisk_log_size":8918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25319026,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8916,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":400095828,"num_objects":48845,"num_object_clones":0,"num_object_copies":195380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141862,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4949,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":799772672,"data_stored":799772672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141889,"ondisk_log_size":141889,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502428,"kb_used_data":196240,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93869412,"statfs":{"total":96636764160,"available":96122277888,"internally_reserved":0,"allocated":200949760,"data_stored":200774294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":3.839}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":2.9399999999999999},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.40500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493556,"kb_used_data":196200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93878284,"statfs":{"total":96636764160,"available":96131362816,"internally_reserved":0,"allocated":200908800,"data_stored":200736480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,7,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.26300000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.374}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.35399999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505564,"kb_used_data":195212,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93866276,"statfs":{"total":96636764160,"available":96119066624,"internally_reserved":0,"allocated":199897088,"data_stored":199721622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,3,7,2,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":4.476},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.23200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":4.375}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":4.2930000000000001},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":3.9990000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498204,"kb_used_data":196240,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93873636,"statfs":{"total":96636764160,"available":96126603264,"internally_reserved":0,"allocated":200949760,"data_stored":200774294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.091},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.498}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":1.464},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":3.1080000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.45300000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":200200192,"data_stored":200200192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199405568,"data_stored":199405568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200015872,"data_stored":200015872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200151040,"data_stored":200151040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:11.770 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:13.611 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:13.651 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:13.696 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-15T03:13:12.859503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":400095828,"num_objects":48845,"num_object_clones":0,"num_object_copies":195380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141862,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4949,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141889,"ondisk_log_size":141889,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1999752,"kb_used_data":783892,"kb_used_omap":14,"kb_used_meta":1215793,"kb_avail":375487608,"statfs":{"total":386547056640,"available":384499310592,"internally_reserved":0,"allocated":802705408,"data_stored":802006690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1244972059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,10,6,16,8,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16277504,"num_objects":-1987,"num_object_clones":0,"num_object_copies":-7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003040"},"pg_stats":[{"pgid":"1.f","version":"35'8816","reported_seq":8882,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.110665+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:09.110665+0000","last_peered":"2024-09-15T03:13:09.110665+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:09.110665+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25346048,"num_objects":3094,"num_object_clones":0,"num_object_copies":12376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8548","reported_seq":8641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.014759+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:09.014759+0000","last_peered":"2024-09-15T03:13:09.014759+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:09.014759+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8548,"log_dups_size":0,"ondisk_log_size":8548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23574130,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":894}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9116","reported_seq":9224,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.279146+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:09.279146+0000","last_peered":"2024-09-15T03:13:09.279146+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:09.279146+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9116,"log_dups_size":0,"ondisk_log_size":9116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25426769,"num_objects":3105,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9115,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8581","reported_seq":12552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:10.192484+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:10.192484+0000","last_peered":"2024-09-15T03:13:10.192484+0000","last_clean":"2024-09-15T03:13:10.192484+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:10.192484+0000","last_undegraded":"2024-09-15T03:13:10.192484+0000","last_fullsized":"2024-09-15T03:13:10.192484+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24958578,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8579,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'8993","reported_seq":9099,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.114619+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:09.114619+0000","last_peered":"2024-09-15T03:13:09.114619+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:09.114619+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8993,"log_dups_size":0,"ondisk_log_size":8993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25294450,"num_objects":3088,"num_object_clones":0,"num_object_copies":12352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'8949","reported_seq":9032,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.159454+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:09.159454+0000","last_peered":"2024-09-15T03:13:09.159454+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:09.159454+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8949,"log_dups_size":0,"ondisk_log_size":8949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24977408,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8949,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1011}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8909","reported_seq":9030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:08.996706+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:08.996706+0000","last_peered":"2024-09-15T03:13:08.996706+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:08.996706+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8909,"log_dups_size":0,"ondisk_log_size":8909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24974962,"num_objects":3049,"num_object_clones":0,"num_object_copies":12196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9010","reported_seq":9127,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.047536+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:09.047536+0000","last_peered":"2024-09-15T03:13:09.047536+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:09.047536+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9010,"log_dups_size":0,"ondisk_log_size":9010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25187954,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9008,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8888","reported_seq":8979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.059779+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:09.059779+0000","last_peered":"2024-09-15T03:13:09.059779+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:09.059779+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8888,"log_dups_size":0,"ondisk_log_size":8888,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25242852,"num_objects":3082,"num_object_clones":0,"num_object_copies":12328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8907","reported_seq":9026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:08.900695+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:08.900695+0000","last_peered":"2024-09-15T03:13:08.900695+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:08.900695+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8907,"log_dups_size":0,"ondisk_log_size":8907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25073266,"num_objects":3061,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8905,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1019}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8914","reported_seq":9007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.118505+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:09.118505+0000","last_peered":"2024-09-15T03:13:09.118505+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:09.118505+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24318379,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":907}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8829","reported_seq":12870,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:10.315938+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:10.315938+0000","last_peered":"2024-09-15T03:13:10.315938+0000","last_clean":"2024-09-15T03:13:10.315938+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:10.315938+0000","last_undegraded":"2024-09-15T03:13:10.315938+0000","last_fullsized":"2024-09-15T03:13:10.315938+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25148217,"num_objects":3070,"num_object_clones":0,"num_object_copies":12280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'8976","reported_seq":10564,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.363471+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:09.363471+0000","last_peered":"2024-09-15T03:13:09.363471+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:09.363471+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25860921,"num_objects":3157,"num_object_clones":0,"num_object_copies":12628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8975,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8706","reported_seq":8806,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:08.851061+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:08.851061+0000","last_peered":"2024-09-15T03:13:08.851061+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:08.851061+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8706,"log_dups_size":0,"ondisk_log_size":8706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24376946,"num_objects":2976,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8704,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":963}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8829","reported_seq":8936,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.053968+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:09.053968+0000","last_peered":"2024-09-15T03:13:09.053968+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:09.053968+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8829,"log_dups_size":0,"ondisk_log_size":8829,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25015922,"num_objects":3054,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8827,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1077}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8918","reported_seq":9031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:09.044818+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:09.044818+0000","last_peered":"2024-09-15T03:13:09.044818+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:09.044818+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8918,"log_dups_size":0,"ondisk_log_size":8918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25319026,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8916,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1032}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":400095828,"num_objects":48845,"num_object_clones":0,"num_object_copies":195380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141862,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4949,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":799772672,"data_stored":799772672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141889,"ondisk_log_size":141889,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":502428,"kb_used_data":196240,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93869412,"statfs":{"total":96636764160,"available":96122277888,"internally_reserved":0,"allocated":200949760,"data_stored":200774294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":3.839}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":2.9399999999999999},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.40500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493556,"kb_used_data":196200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93878284,"statfs":{"total":96636764160,"available":96131362816,"internally_reserved":0,"allocated":200908800,"data_stored":200736480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,7,4,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.26300000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.374}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.35399999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505564,"kb_used_data":195212,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93866276,"statfs":{"total":96636764160,"available":96119066624,"internally_reserved":0,"allocated":199897088,"data_stored":199721622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,3,7,2,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":4.476},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.23200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":4.375}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":4.2930000000000001},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":3.9990000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498204,"kb_used_data":196240,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93873636,"statfs":{"total":96636764160,"available":96126603264,"internally_reserved":0,"allocated":200949760,"data_stored":200774294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.091},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.498}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":1.464},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":3.1080000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.45300000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":200200192,"data_stored":200200192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199405568,"data_stored":199405568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200015872,"data_stored":200015872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200151040,"data_stored":200151040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:13.698 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:13:13.698 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8816", 'reported_seq': 8882, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.110665+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:13:09.110665+0000', 'last_peered': '2024-09-15T03:13:09.110665+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:13:09.110665+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8816, 'log_dups_size': 0, 'ondisk_log_size': 8816, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25346048, 'num_objects': 3094, 'num_object_clones': 0, 'num_object_copies': 12376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8816, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1128}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:13.698 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:13:13.698 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8548", 'reported_seq': 8641, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.014759+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:13:09.014759+0000', 'last_peered': '2024-09-15T03:13:09.014759+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:13:09.014759+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8548, 'log_dups_size': 0, 'ondisk_log_size': 8548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23574130, 'num_objects': 2878, 'num_object_clones': 0, 'num_object_copies': 11512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8546, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 894}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:13.698 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:13:13.699 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9116", 'reported_seq': 9224, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.279146+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:13:09.279146+0000', 'last_peered': '2024-09-15T03:13:09.279146+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:13:09.279146+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9116, 'log_dups_size': 0, 'ondisk_log_size': 9116, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25426769, 'num_objects': 3105, 'num_object_clones': 0, 'num_object_copies': 12420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3105, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9115, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1014}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:13.699 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:13:13.699 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'8993", 'reported_seq': 9099, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.114619+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:13:09.114619+0000', 'last_peered': '2024-09-15T03:13:09.114619+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:13:09.114619+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8993, 'log_dups_size': 0, 'ondisk_log_size': 8993, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25294450, 'num_objects': 3088, 'num_object_clones': 0, 'num_object_copies': 12352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8991, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:13.699 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:13:13.699 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'8949", 'reported_seq': 9032, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.159454+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:13:09.159454+0000', 'last_peered': '2024-09-15T03:13:09.159454+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:13:09.159454+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8949, 'log_dups_size': 0, 'ondisk_log_size': 8949, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24977408, 'num_objects': 3049, 'num_object_clones': 0, 'num_object_copies': 12196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8949, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1011}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:13.699 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:13:13.699 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'8909", 'reported_seq': 9030, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:08.996706+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:13:08.996706+0000', 'last_peered': '2024-09-15T03:13:08.996706+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:13:08.996706+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8909, 'log_dups_size': 0, 'ondisk_log_size': 8909, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24974962, 'num_objects': 3049, 'num_object_clones': 0, 'num_object_copies': 12196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8907, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1023}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:13.699 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:13:13.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9010", 'reported_seq': 9127, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.047536+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:13:09.047536+0000', 'last_peered': '2024-09-15T03:13:09.047536+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:13:09.047536+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9010, 'log_dups_size': 0, 'ondisk_log_size': 9010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25187954, 'num_objects': 3075, 'num_object_clones': 0, 'num_object_copies': 12300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9008, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:13.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:13:13.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8888", 'reported_seq': 8979, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.059779+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:13:09.059779+0000', 'last_peered': '2024-09-15T03:13:09.059779+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:13:09.059779+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8888, 'log_dups_size': 0, 'ondisk_log_size': 8888, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25242852, 'num_objects': 3082, 'num_object_clones': 0, 'num_object_copies': 12328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8885, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1055}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:13.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:13:13.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'8907", 'reported_seq': 9026, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:08.900695+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:13:08.900695+0000', 'last_peered': '2024-09-15T03:13:08.900695+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:13:08.900695+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8907, 'log_dups_size': 0, 'ondisk_log_size': 8907, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25073266, 'num_objects': 3061, 'num_object_clones': 0, 'num_object_copies': 12244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8905, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1019}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:13.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:13:13.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'8914", 'reported_seq': 9007, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.118505+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:13:09.118505+0000', 'last_peered': '2024-09-15T03:13:09.118505+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:13:09.118505+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8914, 'log_dups_size': 0, 'ondisk_log_size': 8914, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24318379, 'num_objects': 2969, 'num_object_clones': 0, 'num_object_copies': 11876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8911, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 907}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:13.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:13:13.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'8976", 'reported_seq': 10564, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.363471+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:13:09.363471+0000', 'last_peered': '2024-09-15T03:13:09.363471+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:13:09.363471+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8976, 'log_dups_size': 0, 'ondisk_log_size': 8976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25860921, 'num_objects': 3157, 'num_object_clones': 0, 'num_object_copies': 12628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8975, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 754, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1870}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:13.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:13:13.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8706", 'reported_seq': 8806, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:08.851061+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:13:08.851061+0000', 'last_peered': '2024-09-15T03:13:08.851061+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:13:08.851061+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8706, 'log_dups_size': 0, 'ondisk_log_size': 8706, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24376946, 'num_objects': 2976, 'num_object_clones': 0, 'num_object_copies': 11904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8704, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:13.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:13:13.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8829", 'reported_seq': 8936, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.053968+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:13:09.053968+0000', 'last_peered': '2024-09-15T03:13:09.053968+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:13:09.053968+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8829, 'log_dups_size': 0, 'ondisk_log_size': 8829, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25015922, 'num_objects': 3054, 'num_object_clones': 0, 'num_object_copies': 12216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8827, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1077}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:13.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:13:13.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'8918", 'reported_seq': 9031, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:09.044818+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:13:09.044818+0000', 'last_peered': '2024-09-15T03:13:09.044818+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:13:09.044818+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8918, 'log_dups_size': 0, 'ondisk_log_size': 8918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25319026, 'num_objects': 3091, 'num_object_clones': 0, 'num_object_copies': 12364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8916, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1032}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:13.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:13:15.517 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:15.517 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27689/191948 objects degraded (14.425%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":518,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47987,"data_bytes":393067092,"bytes_used":2052689920,"bytes_avail":384494366720,"bytes_total":386547056640,"degraded_objects":27689,"degraded_total":191948,"degraded_ratio":0.14425261008189719,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":237},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:13:15.629 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:13:15.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:17.153 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:17.163 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:17.174 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-15T03:13:16.860535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":392034900,"num_objects":47861,"num_object_clones":0,"num_object_copies":191444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47861,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142846,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4995,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142873,"ondisk_log_size":142873,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1988200,"kb_used_data":768176,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375499160,"statfs":{"total":386547056640,"available":384511139840,"internally_reserved":0,"allocated":786612224,"data_stored":785913506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,5,11,14,11,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17260544,"num_objects":-2107,"num_object_clones":0,"num_object_copies":-8428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2107,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003065"},"pg_stats":[{"pgid":"1.f","version":"35'8877","reported_seq":8943,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.174115+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:14.174115+0000","last_peered":"2024-09-15T03:13:14.174115+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:14.174115+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8877,"log_dups_size":0,"ondisk_log_size":8877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24846336,"num_objects":3033,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8603","reported_seq":8697,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:13.797208+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:13.797208+0000","last_peered":"2024-09-15T03:13:13.797208+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:13.797208+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8603,"log_dups_size":0,"ondisk_log_size":8603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23123570,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8601,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":839}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9171","reported_seq":9279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.320994+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:14.320994+0000","last_peered":"2024-09-15T03:13:14.320994+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:14.320994+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9171,"log_dups_size":0,"ondisk_log_size":9171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24976209,"num_objects":3050,"num_object_clones":0,"num_object_copies":12200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9170,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":959}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8637","reported_seq":12609,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:15.222165+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:15.222165+0000","last_peered":"2024-09-15T03:13:15.222165+0000","last_clean":"2024-09-15T03:13:15.222165+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:15.222165+0000","last_undegraded":"2024-09-15T03:13:15.222165+0000","last_fullsized":"2024-09-15T03:13:15.222165+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8637,"log_dups_size":0,"ondisk_log_size":8637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24499826,"num_objects":2991,"num_object_clones":0,"num_object_copies":11964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8635,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9053","reported_seq":9159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.242702+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:14.242702+0000","last_peered":"2024-09-15T03:13:14.242702+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:14.242702+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9053,"log_dups_size":0,"ondisk_log_size":9053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24802930,"num_objects":3028,"num_object_clones":0,"num_object_copies":12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9014","reported_seq":9098,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:13.984572+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:13.984572+0000","last_peered":"2024-09-15T03:13:13.984572+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:13.984572+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9014,"log_dups_size":0,"ondisk_log_size":9014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444928,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9014,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'8970","reported_seq":9088,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.138068+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:14.138068+0000","last_peered":"2024-09-15T03:13:14.138068+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:14.138068+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8970,"log_dups_size":0,"ondisk_log_size":8970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24475250,"num_objects":2988,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8968,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":962}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9086","reported_seq":9203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.099379+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:14.099379+0000","last_peered":"2024-09-15T03:13:14.099379+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:14.099379+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9086,"log_dups_size":0,"ondisk_log_size":9086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24565362,"num_objects":2999,"num_object_clones":0,"num_object_copies":11996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9084,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8937","reported_seq":9028,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.167597+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:14.167597+0000","last_peered":"2024-09-15T03:13:14.167597+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:14.167597+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8937,"log_dups_size":0,"ondisk_log_size":8937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24841444,"num_objects":3033,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8934,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'8972","reported_seq":9091,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.323631+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:14.323631+0000","last_peered":"2024-09-15T03:13:14.323631+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:14.323631+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8972,"log_dups_size":0,"ondisk_log_size":8972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24540786,"num_objects":2996,"num_object_clones":0,"num_object_copies":11984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8970,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":954}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'8990","reported_seq":9084,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.153521+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:14.153521+0000","last_peered":"2024-09-15T03:13:14.153521+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:14.153521+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8990,"log_dups_size":0,"ondisk_log_size":8990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23695787,"num_objects":2893,"num_object_clones":0,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8987,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8899","reported_seq":12939,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:15.344103+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:15.344103+0000","last_peered":"2024-09-15T03:13:15.344103+0000","last_clean":"2024-09-15T03:13:15.344103+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:15.344103+0000","last_undegraded":"2024-09-15T03:13:15.344103+0000","last_fullsized":"2024-09-15T03:13:15.344103+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8899,"log_dups_size":0,"ondisk_log_size":8899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24574777,"num_objects":3000,"num_object_clones":0,"num_object_copies":12000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8898,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9038","reported_seq":10718,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.411392+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:14.411392+0000","last_peered":"2024-09-15T03:13:14.411392+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:14.411392+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9038,"log_dups_size":0,"ondisk_log_size":9038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25353017,"num_objects":3095,"num_object_clones":0,"num_object_copies":12380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":800,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8765","reported_seq":8865,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.042191+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:14.042191+0000","last_peered":"2024-09-15T03:13:14.042191+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:14.042191+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8765,"log_dups_size":0,"ondisk_log_size":8765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23893618,"num_objects":2917,"num_object_clones":0,"num_object_copies":11668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8763,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":904}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8886","reported_seq":8993,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.373766+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:14.373766+0000","last_peered":"2024-09-15T03:13:14.373766+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:14.373766+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8886,"log_dups_size":0,"ondisk_log_size":8886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24548978,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8884,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1020}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'8975","reported_seq":9086,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:14.007521+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:14.007521+0000","last_peered":"2024-09-15T03:13:14.007521+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:14.007521+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24852082,"num_objects":3034,"num_object_clones":0,"num_object_copies":12136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8973,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":392034900,"num_objects":47861,"num_object_clones":0,"num_object_copies":191444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47861,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142846,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4995,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":783704064,"data_stored":783704064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142873,"ondisk_log_size":142873,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506776,"kb_used_data":192272,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93865064,"statfs":{"total":96636764160,"available":96117825536,"internally_reserved":0,"allocated":196886528,"data_stored":196711062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,7,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.19600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":3.6339999999999999,"5min":2.6280000000000001,"15min":1.964},"min":{"1min":0.25700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":40.710999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.7070000000000001,"15min":1.355},"min":{"1min":0.24099999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.5839999999999996,"5min":13.343999999999999,"15min":13.343999999999999},"last":0.42799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":4.5170000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493740,"kb_used_data":192224,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93878100,"statfs":{"total":96636764160,"available":96131174400,"internally_reserved":0,"allocated":196837376,"data_stored":196665056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.29999999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:12:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.617,"15min":1.2410000000000001},"min":{"1min":0.20499999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.0790000000000006,"5min":19.343,"15min":19.343},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.71,"15min":1.296},"min":{"1min":0.25700000000000001,"5min":0.23699999999999999,"15min":0.20999999999999999},"max":{"1min":8.1400000000000006,"5min":23.311,"15min":23.311},"last":0.35499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.373}]}]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489184,"kb_used_data":191320,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93882656,"statfs":{"total":96636764160,"available":96135839744,"internally_reserved":0,"allocated":195911680,"data_stored":195736214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,3,7,2,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":4.476},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.23200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":4.375}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":4.2930000000000001},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":3.9990000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498500,"kb_used_data":192360,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93873340,"statfs":{"total":96636764160,"available":96126300160,"internally_reserved":0,"allocated":196976640,"data_stored":196801174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":1.091},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.498}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":1.464},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":3.1080000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.45300000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":196186112,"data_stored":196186112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195391488,"data_stored":195391488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195981312,"data_stored":195981312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":196145152,"data_stored":196145152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:20.176 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:21.968 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:22.004 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:22.061 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-15T03:13:20.863583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":384170580,"num_objects":46901,"num_object_clones":0,"num_object_copies":187604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143806,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5061,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143833,"ondisk_log_size":143833,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1972980,"kb_used_data":752980,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375514380,"statfs":{"total":386547056640,"available":384526725120,"internally_reserved":0,"allocated":771051520,"data_stored":770352802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,9,4,14,11,16,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24059904,"num_objects":-2937,"num_object_clones":0,"num_object_copies":-11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2937,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005048"},"pg_stats":[{"pgid":"1.f","version":"35'8929","reported_seq":8995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.149329+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:19.149329+0000","last_peered":"2024-09-15T03:13:19.149329+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:19.149329+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8929,"log_dups_size":0,"ondisk_log_size":8929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24420352,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8929,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1015}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8649","reported_seq":8743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.044489+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:19.044489+0000","last_peered":"2024-09-15T03:13:19.044489+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:19.044489+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22746738,"num_objects":2777,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9227","reported_seq":9335,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.250438+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:19.250438+0000","last_peered":"2024-09-15T03:13:19.250438+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:19.250438+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24517457,"num_objects":2994,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9226,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8691","reported_seq":12663,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:20.376615+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:20.376615+0000","last_peered":"2024-09-15T03:13:20.376615+0000","last_clean":"2024-09-15T03:13:20.376615+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:20.376615+0000","last_undegraded":"2024-09-15T03:13:20.376615+0000","last_fullsized":"2024-09-15T03:13:20.376615+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8691,"log_dups_size":0,"ondisk_log_size":8691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24057458,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9111","reported_seq":9217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:18.954380+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:18.954380+0000","last_peered":"2024-09-15T03:13:18.954380+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:18.954380+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9111,"log_dups_size":0,"ondisk_log_size":9111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24327794,"num_objects":2970,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9109,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9092","reported_seq":9174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.245999+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:19.245999+0000","last_peered":"2024-09-15T03:13:19.245999+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:19.245999+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9092,"log_dups_size":0,"ondisk_log_size":9092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805952,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9033","reported_seq":9154,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:18.899883+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:18.899883+0000","last_peered":"2024-09-15T03:13:18.899883+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:18.899883+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9033,"log_dups_size":0,"ondisk_log_size":9033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23959154,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9031,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":899}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9142","reported_seq":9259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.122110+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:19.122110+0000","last_peered":"2024-09-15T03:13:19.122110+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:19.122110+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24106610,"num_objects":2943,"num_object_clones":0,"num_object_copies":11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8992","reported_seq":9083,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.289602+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:19.289602+0000","last_peered":"2024-09-15T03:13:19.289602+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:19.289602+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24390884,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8989,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":951}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9036","reported_seq":9155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.344386+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:19.344386+0000","last_peered":"2024-09-15T03:13:19.344386+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:19.344386+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016498,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9034,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":890}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9051","reported_seq":9145,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:18.979167+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:18.979167+0000","last_peered":"2024-09-15T03:13:18.979167+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:18.979167+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9051,"log_dups_size":0,"ondisk_log_size":9051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23196075,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9048,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8962","reported_seq":13003,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:20.474157+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:20.474157+0000","last_peered":"2024-09-15T03:13:20.474157+0000","last_clean":"2024-09-15T03:13:20.474157+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:20.474157+0000","last_undegraded":"2024-09-15T03:13:20.474157+0000","last_fullsized":"2024-09-15T03:13:20.474157+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24058681,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9088","reported_seq":10900,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.353450+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:19.353450+0000","last_peered":"2024-09-15T03:13:19.353450+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:19.353450+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9088,"log_dups_size":0,"ondisk_log_size":9088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24943417,"num_objects":3045,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9087,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8828","reported_seq":8927,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.166043+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:19.166043+0000","last_peered":"2024-09-15T03:13:19.166043+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:19.166043+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8828,"log_dups_size":0,"ondisk_log_size":8828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23377522,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8826,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8960","reported_seq":9067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.193930+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:19.193930+0000","last_peered":"2024-09-15T03:13:19.193930+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:19.193930+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8960,"log_dups_size":0,"ondisk_log_size":8960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23942770,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":946}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9042","reported_seq":9155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.112398+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:19.112398+0000","last_peered":"2024-09-15T03:13:19.112398+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:19.112398+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24303218,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":384170580,"num_objects":46901,"num_object_clones":0,"num_object_copies":187604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143806,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5061,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":767795200,"data_stored":767795200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143833,"ondisk_log_size":143833,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490480,"kb_used_data":188456,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93881360,"statfs":{"total":96636764160,"available":96134512640,"internally_reserved":0,"allocated":192978944,"data_stored":192803478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,7,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.35499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":2.653},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":2.835}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.50900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494044,"kb_used_data":188372,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93877796,"statfs":{"total":96636764160,"available":96130863104,"internally_reserved":0,"allocated":192892928,"data_stored":192720608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.28100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.308},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.34399999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.373},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.38600000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489552,"kb_used_data":187528,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93882288,"statfs":{"total":96636764160,"available":96135462912,"internally_reserved":0,"allocated":192028672,"data_stored":191853206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":16.010999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":49.399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":12.371},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":2.177}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":59.225999999999999},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":12.102}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498904,"kb_used_data":188624,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93872936,"statfs":{"total":96636764160,"available":96125886464,"internally_reserved":0,"allocated":193150976,"data_stored":192975510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.3120000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":1.194},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.42799999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.28699999999999998},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192225280,"data_stored":192225280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":191401984,"data_stored":191401984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192016384,"data_stored":192016384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192151552,"data_stored":192151552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:22.062 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:23.727 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:23.790 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-15T03:13:22.864106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":384170580,"num_objects":46901,"num_object_clones":0,"num_object_copies":187604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143806,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5061,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143833,"ondisk_log_size":143833,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1972980,"kb_used_data":752980,"kb_used_omap":14,"kb_used_meta":1219953,"kb_avail":375514380,"statfs":{"total":386547056640,"available":384526725120,"internally_reserved":0,"allocated":771051520,"data_stored":770352802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1249231899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,9,4,14,11,16,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15925248,"num_objects":-1944,"num_object_clones":0,"num_object_copies":-7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005131"},"pg_stats":[{"pgid":"1.f","version":"35'8929","reported_seq":8995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.149329+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:19.149329+0000","last_peered":"2024-09-15T03:13:19.149329+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:19.149329+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8929,"log_dups_size":0,"ondisk_log_size":8929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24420352,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8929,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1015}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8649","reported_seq":8743,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.044489+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:19.044489+0000","last_peered":"2024-09-15T03:13:19.044489+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:19.044489+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22746738,"num_objects":2777,"num_object_clones":0,"num_object_copies":11108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8647,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":793}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9227","reported_seq":9335,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.250438+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:19.250438+0000","last_peered":"2024-09-15T03:13:19.250438+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:19.250438+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24517457,"num_objects":2994,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9226,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":903}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8691","reported_seq":12663,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:20.376615+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:20.376615+0000","last_peered":"2024-09-15T03:13:20.376615+0000","last_clean":"2024-09-15T03:13:20.376615+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:20.376615+0000","last_undegraded":"2024-09-15T03:13:20.376615+0000","last_fullsized":"2024-09-15T03:13:20.376615+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8691,"log_dups_size":0,"ondisk_log_size":8691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24057458,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9111","reported_seq":9217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:18.954380+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:18.954380+0000","last_peered":"2024-09-15T03:13:18.954380+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:18.954380+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9111,"log_dups_size":0,"ondisk_log_size":9111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24327794,"num_objects":2970,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9109,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":886}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9092","reported_seq":9174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.245999+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:19.245999+0000","last_peered":"2024-09-15T03:13:19.245999+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:19.245999+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9092,"log_dups_size":0,"ondisk_log_size":9092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805952,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":868}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9033","reported_seq":9154,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:18.899883+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:18.899883+0000","last_peered":"2024-09-15T03:13:18.899883+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:18.899883+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9033,"log_dups_size":0,"ondisk_log_size":9033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23959154,"num_objects":2925,"num_object_clones":0,"num_object_copies":11700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9031,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":899}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9142","reported_seq":9259,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.122110+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:19.122110+0000","last_peered":"2024-09-15T03:13:19.122110+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:19.122110+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24106610,"num_objects":2943,"num_object_clones":0,"num_object_copies":11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'8992","reported_seq":9083,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.289602+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:19.289602+0000","last_peered":"2024-09-15T03:13:19.289602+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:19.289602+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24390884,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8989,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":951}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9036","reported_seq":9155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.344386+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:19.344386+0000","last_peered":"2024-09-15T03:13:19.344386+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:19.344386+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24016498,"num_objects":2932,"num_object_clones":0,"num_object_copies":11728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9034,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":890}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9051","reported_seq":9145,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:18.979167+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:18.979167+0000","last_peered":"2024-09-15T03:13:18.979167+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:18.979167+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9051,"log_dups_size":0,"ondisk_log_size":9051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23196075,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9048,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":770}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'8962","reported_seq":13003,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:20.474157+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:20.474157+0000","last_peered":"2024-09-15T03:13:20.474157+0000","last_clean":"2024-09-15T03:13:20.474157+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:20.474157+0000","last_undegraded":"2024-09-15T03:13:20.474157+0000","last_fullsized":"2024-09-15T03:13:20.474157+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24058681,"num_objects":2937,"num_object_clones":0,"num_object_copies":11748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9088","reported_seq":10900,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.353450+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:19.353450+0000","last_peered":"2024-09-15T03:13:19.353450+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:19.353450+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9088,"log_dups_size":0,"ondisk_log_size":9088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24943417,"num_objects":3045,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9087,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":866,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1870}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8828","reported_seq":8927,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.166043+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:19.166043+0000","last_peered":"2024-09-15T03:13:19.166043+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:19.166043+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8828,"log_dups_size":0,"ondisk_log_size":8828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23377522,"num_objects":2854,"num_object_clones":0,"num_object_copies":11416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8826,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'8960","reported_seq":9067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.193930+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:19.193930+0000","last_peered":"2024-09-15T03:13:19.193930+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:19.193930+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8960,"log_dups_size":0,"ondisk_log_size":8960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23942770,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":946}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9042","reported_seq":9155,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:19.112398+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:19.112398+0000","last_peered":"2024-09-15T03:13:19.112398+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:19.112398+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24303218,"num_objects":2967,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":908}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":384170580,"num_objects":46901,"num_object_clones":0,"num_object_copies":187604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143806,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5061,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":767795200,"data_stored":767795200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143833,"ondisk_log_size":143833,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490480,"kb_used_data":188456,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93881360,"statfs":{"total":96636764160,"available":96134512640,"internally_reserved":0,"allocated":192978944,"data_stored":192803478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,7,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.35499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":2.653},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":2.835}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.50900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494044,"kb_used_data":188372,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93877796,"statfs":{"total":96636764160,"available":96130863104,"internally_reserved":0,"allocated":192892928,"data_stored":192720608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,2,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.28100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.308},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.34399999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.373},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.38600000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489552,"kb_used_data":187528,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93882288,"statfs":{"total":96636764160,"available":96135462912,"internally_reserved":0,"allocated":192028672,"data_stored":191853206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":16.010999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":49.399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":12.371},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":2.177}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":59.225999999999999},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":12.102}]}]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498904,"kb_used_data":188624,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93872936,"statfs":{"total":96636764160,"available":96125886464,"internally_reserved":0,"allocated":193150976,"data_stored":192975510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,3,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:12:20 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9379999999999999,"15min":1.1479999999999999},"min":{"1min":0.19400000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":7.9470000000000001,"5min":19.527000000000001,"15min":19.527000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":3.302,"5min":3.1080000000000001,"15min":2.0489999999999999},"min":{"1min":0.27000000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":19.765999999999998,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.3120000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":1.194},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.42799999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.28699999999999998},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192225280,"data_stored":192225280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":191401984,"data_stored":191401984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192016384,"data_stored":192016384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192151552,"data_stored":192151552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:23.791 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:13:23.791 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'8929", 'reported_seq': 8995, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.149329+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:13:19.149329+0000', 'last_peered': '2024-09-15T03:13:19.149329+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:13:19.149329+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8929, 'log_dups_size': 0, 'ondisk_log_size': 8929, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24420352, 'num_objects': 2981, 'num_object_clones': 0, 'num_object_copies': 11924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8929, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1015}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8649", 'reported_seq': 8743, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.044489+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:13:19.044489+0000', 'last_peered': '2024-09-15T03:13:19.044489+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:13:19.044489+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8649, 'log_dups_size': 0, 'ondisk_log_size': 8649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22746738, 'num_objects': 2777, 'num_object_clones': 0, 'num_object_copies': 11108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8647, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 793}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9227", 'reported_seq': 9335, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.250438+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:13:19.250438+0000', 'last_peered': '2024-09-15T03:13:19.250438+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:13:19.250438+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9227, 'log_dups_size': 0, 'ondisk_log_size': 9227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24517457, 'num_objects': 2994, 'num_object_clones': 0, 'num_object_copies': 11976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2994, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9226, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 903}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9111", 'reported_seq': 9217, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:18.954380+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:13:18.954380+0000', 'last_peered': '2024-09-15T03:13:18.954380+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:13:18.954380+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9111, 'log_dups_size': 0, 'ondisk_log_size': 9111, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24327794, 'num_objects': 2970, 'num_object_clones': 0, 'num_object_copies': 11880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9109, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 886}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9092", 'reported_seq': 9174, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.245999+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:13:19.245999+0000', 'last_peered': '2024-09-15T03:13:19.245999+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:13:19.245999+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9092, 'log_dups_size': 0, 'ondisk_log_size': 9092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23805952, 'num_objects': 2906, 'num_object_clones': 0, 'num_object_copies': 11624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9092, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 868}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:13:23.792 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9033", 'reported_seq': 9154, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:18.899883+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:13:18.899883+0000', 'last_peered': '2024-09-15T03:13:18.899883+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:13:18.899883+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9033, 'log_dups_size': 0, 'ondisk_log_size': 9033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23959154, 'num_objects': 2925, 'num_object_clones': 0, 'num_object_copies': 11700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9031, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 899}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9142", 'reported_seq': 9259, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.122110+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:13:19.122110+0000', 'last_peered': '2024-09-15T03:13:19.122110+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:13:19.122110+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9142, 'log_dups_size': 0, 'ondisk_log_size': 9142, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24106610, 'num_objects': 2943, 'num_object_clones': 0, 'num_object_copies': 11772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9140, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'8992", 'reported_seq': 9083, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.289602+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:13:19.289602+0000', 'last_peered': '2024-09-15T03:13:19.289602+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:13:19.289602+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8992, 'log_dups_size': 0, 'ondisk_log_size': 8992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24390884, 'num_objects': 2978, 'num_object_clones': 0, 'num_object_copies': 11912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8989, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 951}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9036", 'reported_seq': 9155, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.344386+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:13:19.344386+0000', 'last_peered': '2024-09-15T03:13:19.344386+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:13:19.344386+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9036, 'log_dups_size': 0, 'ondisk_log_size': 9036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24016498, 'num_objects': 2932, 'num_object_clones': 0, 'num_object_copies': 11728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9034, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 890}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9051", 'reported_seq': 9145, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:18.979167+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:13:18.979167+0000', 'last_peered': '2024-09-15T03:13:18.979167+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:13:18.979167+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9051, 'log_dups_size': 0, 'ondisk_log_size': 9051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23196075, 'num_objects': 2832, 'num_object_clones': 0, 'num_object_copies': 11328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9048, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 770}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:13:23.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9088", 'reported_seq': 10900, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.353450+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:13:19.353450+0000', 'last_peered': '2024-09-15T03:13:19.353450+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:13:19.353450+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9088, 'log_dups_size': 0, 'ondisk_log_size': 9088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24943417, 'num_objects': 3045, 'num_object_clones': 0, 'num_object_copies': 12180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1175, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3045, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9087, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 866, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1870}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:23.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:13:23.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8828", 'reported_seq': 8927, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.166043+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:13:19.166043+0000', 'last_peered': '2024-09-15T03:13:19.166043+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:13:19.166043+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8828, 'log_dups_size': 0, 'ondisk_log_size': 8828, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23377522, 'num_objects': 2854, 'num_object_clones': 0, 'num_object_copies': 11416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8826, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 841}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:23.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:13:23.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'8960", 'reported_seq': 9067, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.193930+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:13:19.193930+0000', 'last_peered': '2024-09-15T03:13:19.193930+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:13:19.193930+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8960, 'log_dups_size': 0, 'ondisk_log_size': 8960, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23942770, 'num_objects': 2923, 'num_object_clones': 0, 'num_object_copies': 11692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8958, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 946}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:23.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:13:23.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9042", 'reported_seq': 9155, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:19.112398+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:13:19.112398+0000', 'last_peered': '2024-09-15T03:13:19.112398+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:13:19.112398+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9042, 'log_dups_size': 0, 'ondisk_log_size': 9042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24303218, 'num_objects': 2967, 'num_object_clones': 0, 'num_object_copies': 11868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9040, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 908}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:23.794 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:13:25.561 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:25.561 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27575/184212 objects degraded (14.969%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":528,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":46053,"data_bytes":377223764,"bytes_used":2004045824,"bytes_avail":384543010816,"bytes_total":386547056640,"degraded_objects":27575,"degraded_total":184212,"degraded_ratio":0.14969165960958028,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":232},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T03:07:06.670397+0000","services":{}},"progress_events":{}} 2024-09-15T03:13:25.596 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:13:25.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:27.446 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:27.515 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:27.526 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-09-15T03:13:26.865091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376216148,"num_objects":45930,"num_object_clones":0,"num_object_copies":183720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45930,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144777,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5109,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144804,"ondisk_log_size":144804,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1961440,"kb_used_data":737244,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375525920,"statfs":{"total":386547056640,"available":384538542080,"internally_reserved":0,"allocated":754937856,"data_stored":754239138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,3,14,9,10,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16850944,"num_objects":-2057,"num_object_clones":0,"num_object_copies":-8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2057,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005111"},"pg_stats":[{"pgid":"1.f","version":"35'8991","reported_seq":9057,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.166356+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:24.166356+0000","last_peered":"2024-09-15T03:13:24.166356+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:24.166356+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8991,"log_dups_size":0,"ondisk_log_size":8991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23912448,"num_objects":2919,"num_object_clones":0,"num_object_copies":11676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":953}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8698","reported_seq":8789,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.068499+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:24.068499+0000","last_peered":"2024-09-15T03:13:24.068499+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:24.068499+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8698,"log_dups_size":0,"ondisk_log_size":8698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22345330,"num_objects":2728,"num_object_clones":0,"num_object_copies":10912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9287","reported_seq":9395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.429429+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:24.429429+0000","last_peered":"2024-09-15T03:13:24.429429+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:24.429429+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24025937,"num_objects":2934,"num_object_clones":0,"num_object_copies":11736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2934,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9286,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8740","reported_seq":12712,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:25.203944+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:25.203944+0000","last_peered":"2024-09-15T03:13:25.203944+0000","last_clean":"2024-09-15T03:13:25.203944+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:25.203944+0000","last_undegraded":"2024-09-15T03:13:25.203944+0000","last_fullsized":"2024-09-15T03:13:25.203944+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8740,"log_dups_size":0,"ondisk_log_size":8740,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23656050,"num_objects":2888,"num_object_clones":0,"num_object_copies":11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8738,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9182","reported_seq":9288,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:23.969386+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:23.969386+0000","last_peered":"2024-09-15T03:13:23.969386+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:23.969386+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23746162,"num_objects":2899,"num_object_clones":0,"num_object_copies":11596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9180,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9153","reported_seq":9237,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.243423+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:24.243423+0000","last_peered":"2024-09-15T03:13:24.243423+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:24.243423+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23306240,"num_objects":2845,"num_object_clones":0,"num_object_copies":11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9153,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":807}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9088","reported_seq":9209,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.041207+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:24.041207+0000","last_peered":"2024-09-15T03:13:24.041207+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:24.041207+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9088,"log_dups_size":0,"ondisk_log_size":9088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23508594,"num_objects":2870,"num_object_clones":0,"num_object_copies":11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9086,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":844}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9201","reported_seq":9318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.036415+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:24.036415+0000","last_peered":"2024-09-15T03:13:24.036415+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:24.036415+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9201,"log_dups_size":0,"ondisk_log_size":9201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23623282,"num_objects":2884,"num_object_clones":0,"num_object_copies":11536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":828}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9054","reported_seq":9145,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.260018+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:24.260018+0000","last_peered":"2024-09-15T03:13:24.260018+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:24.260018+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9054,"log_dups_size":0,"ondisk_log_size":9054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23882980,"num_objects":2916,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":889}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9097","reported_seq":9216,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.305154+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:24.305154+0000","last_peered":"2024-09-15T03:13:24.305154+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:24.305154+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23516786,"num_objects":2871,"num_object_clones":0,"num_object_copies":11484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":829}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9117","reported_seq":9211,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.099120+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:24.099120+0000","last_peered":"2024-09-15T03:13:24.099120+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:24.099120+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9117,"log_dups_size":0,"ondisk_log_size":9117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22655403,"num_objects":2766,"num_object_clones":0,"num_object_copies":11064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9114,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9036","reported_seq":13077,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:25.515473+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:25.515473+0000","last_peered":"2024-09-15T03:13:25.515473+0000","last_clean":"2024-09-15T03:13:25.515473+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:25.515473+0000","last_undegraded":"2024-09-15T03:13:25.515473+0000","last_fullsized":"2024-09-15T03:13:25.515473+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23452473,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9035,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9152","reported_seq":11060,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.458881+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:24.458881+0000","last_peered":"2024-09-15T03:13:24.458881+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:24.458881+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9152,"log_dups_size":0,"ondisk_log_size":9152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24419129,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":914,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8879","reported_seq":8979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.085188+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:24.085188+0000","last_peered":"2024-09-15T03:13:24.085188+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:24.085188+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8879,"log_dups_size":0,"ondisk_log_size":8879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22959730,"num_objects":2803,"num_object_clones":0,"num_object_copies":11212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8877,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":790}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9024","reported_seq":9131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.480695+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:24.480695+0000","last_peered":"2024-09-15T03:13:24.480695+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:24.480695+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23418482,"num_objects":2859,"num_object_clones":0,"num_object_copies":11436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":882}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9105","reported_seq":9217,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:24.194210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:24.194210+0000","last_peered":"2024-09-15T03:13:24.194210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:24.194210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23787122,"num_objects":2904,"num_object_clones":0,"num_object_copies":11616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9103,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376216148,"num_objects":45930,"num_object_clones":0,"num_object_copies":183720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45930,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144777,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5109,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":751910912,"data_stored":751910912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144804,"ondisk_log_size":144804,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490752,"kb_used_data":184560,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93881088,"statfs":{"total":96636764160,"available":96134234112,"internally_reserved":0,"allocated":188989440,"data_stored":188813974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":7.585}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.35299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.439},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.38300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494308,"kb_used_data":184476,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93877532,"statfs":{"total":96636764160,"available":96130592768,"internally_reserved":0,"allocated":188903424,"data_stored":188731104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.22500000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":6.8410000000000002},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":6.8739999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.27000000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493916,"kb_used_data":183560,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93877924,"statfs":{"total":96636764160,"available":96130994176,"internally_reserved":0,"allocated":187965440,"data_stored":187789974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,4,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":7.7629999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":3.968},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":7.8570000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":4.4029999999999996},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.38400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482464,"kb_used_data":184648,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93889376,"statfs":{"total":96636764160,"available":96142721024,"internally_reserved":0,"allocated":189079552,"data_stored":188904086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,3,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.66100000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":4.0069999999999997},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.79100000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.46899999999999997},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.36399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188235776,"data_stored":188235776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187441152,"data_stored":187441152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":188047360,"data_stored":188047360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":188186624,"data_stored":188186624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:30.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:32.220 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:32.252 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:32.289 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-15T03:13:30.866050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":368179796,"num_objects":44949,"num_object_clones":0,"num_object_copies":179796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44949,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145758,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145785,"ondisk_log_size":145785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1933468,"kb_used_data":721772,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375553892,"statfs":{"total":386547056640,"available":384567185408,"internally_reserved":0,"allocated":739094528,"data_stored":738395810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,4,7,13,22,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23855104,"num_objects":-2912,"num_object_clones":0,"num_object_copies":-11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2912,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.f","version":"35'9059","reported_seq":9124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.293075+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:29.293075+0000","last_peered":"2024-09-15T03:13:29.293075+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:29.293075+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9059,"log_dups_size":0,"ondisk_log_size":9059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23355392,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8757","reported_seq":8850,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.122079+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:29.122079+0000","last_peered":"2024-09-15T03:13:29.122079+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:29.122079+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21862002,"num_objects":2669,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9348","reported_seq":9455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.399491+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:29.399491+0000","last_peered":"2024-09-15T03:13:29.399491+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:29.399491+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23526225,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9347,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8815","reported_seq":12785,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:30.466519+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:30.466519+0000","last_peered":"2024-09-15T03:13:30.466519+0000","last_clean":"2024-09-15T03:13:30.466519+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:30.466519+0000","last_undegraded":"2024-09-15T03:13:30.466519+0000","last_fullsized":"2024-09-15T03:13:30.466519+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041650,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8813,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9239","reported_seq":9345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.260865+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:29.260865+0000","last_peered":"2024-09-15T03:13:29.260865+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:29.260865+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9239,"log_dups_size":0,"ondisk_log_size":9239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23279218,"num_objects":2842,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9211","reported_seq":9295,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.297528+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:29.297528+0000","last_peered":"2024-09-15T03:13:29.297528+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:29.297528+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9211,"log_dups_size":0,"ondisk_log_size":9211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22831104,"num_objects":2787,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9141","reported_seq":9262,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.216623+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:29.216623+0000","last_peered":"2024-09-15T03:13:29.216623+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:29.216623+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9141,"log_dups_size":0,"ondisk_log_size":9141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23074418,"num_objects":2817,"num_object_clones":0,"num_object_copies":11268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9139,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":791}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9256","reported_seq":9373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.045779+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:29.045779+0000","last_peered":"2024-09-15T03:13:29.045779+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:29.045779+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23172722,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9254,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9105","reported_seq":9196,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.241711+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:29.241711+0000","last_peered":"2024-09-15T03:13:29.241711+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:29.241711+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23465188,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9158","reported_seq":9277,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.084293+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:29.084293+0000","last_peered":"2024-09-15T03:13:29.084293+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:29.084293+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9158,"log_dups_size":0,"ondisk_log_size":9158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23017074,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":768}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9186","reported_seq":9279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.336740+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:29.336740+0000","last_peered":"2024-09-15T03:13:29.336740+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:29.336740+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22090155,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9091","reported_seq":13132,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:30.567785+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:30.567785+0000","last_peered":"2024-09-15T03:13:30.567785+0000","last_clean":"2024-09-15T03:13:30.567785+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:30.567785+0000","last_undegraded":"2024-09-15T03:13:30.567785+0000","last_fullsized":"2024-09-15T03:13:30.567785+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23001913,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9225","reported_seq":11209,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.520539+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:29.520539+0000","last_peered":"2024-09-15T03:13:29.520539+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:29.520539+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9225,"log_dups_size":0,"ondisk_log_size":9225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23821113,"num_objects":2908,"num_object_clones":0,"num_object_copies":11632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1819}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8938","reported_seq":9038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.180912+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:29.180912+0000","last_peered":"2024-09-15T03:13:29.180912+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:29.180912+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8938,"log_dups_size":0,"ondisk_log_size":8938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22476402,"num_objects":2744,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8936,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9091","reported_seq":9198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.413397+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:29.413397+0000","last_peered":"2024-09-15T03:13:29.413397+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:29.413397+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22869618,"num_objects":2792,"num_object_clones":0,"num_object_copies":11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9165","reported_seq":9278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.138448+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:29.138448+0000","last_peered":"2024-09-15T03:13:29.138448+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:29.138448+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23295602,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":368179796,"num_objects":44949,"num_object_clones":0,"num_object_copies":179796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44949,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145758,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":735809536,"data_stored":735809536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145785,"ondisk_log_size":145785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491076,"kb_used_data":180732,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93880764,"statfs":{"total":96636764160,"available":96133902336,"internally_reserved":0,"allocated":185069568,"data_stored":184894102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.246}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.311},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.46800000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.495}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482172,"kb_used_data":180660,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93889668,"statfs":{"total":96636764160,"available":96143020032,"internally_reserved":0,"allocated":184995840,"data_stored":184823520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.25},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.26900000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":7.4669999999999996},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":7.5119999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.34000000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477472,"kb_used_data":179592,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93894368,"statfs":{"total":96636764160,"available":96147832832,"internally_reserved":0,"allocated":183902208,"data_stored":183726742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,3,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":22.803000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.219}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":27.321999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":2.8620000000000001},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.39800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482748,"kb_used_data":180788,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93889092,"statfs":{"total":96636764160,"available":96142430208,"internally_reserved":0,"allocated":185126912,"data_stored":184951446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,5,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.28999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":0.254},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.40699999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.32700000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":4.2789999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184233984,"data_stored":184233984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183410688,"data_stored":183410688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184000512,"data_stored":184000512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184164352,"data_stored":184164352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:32.290 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:34.189 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:34.207 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:34.214 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-09-15T03:13:32.866585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":368179796,"num_objects":44949,"num_object_clones":0,"num_object_copies":179796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44949,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145758,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145785,"ondisk_log_size":145785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1933468,"kb_used_data":721772,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375553892,"statfs":{"total":386547056640,"available":384567185408,"internally_reserved":0,"allocated":739094528,"data_stored":738395810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,4,7,13,22,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15990784,"num_objects":-1952,"num_object_clones":0,"num_object_copies":-7808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1952,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003001"},"pg_stats":[{"pgid":"1.f","version":"35'9059","reported_seq":9124,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.293075+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:29.293075+0000","last_peered":"2024-09-15T03:13:29.293075+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:29.293075+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9059,"log_dups_size":0,"ondisk_log_size":9059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23355392,"num_objects":2851,"num_object_clones":0,"num_object_copies":11404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8757","reported_seq":8850,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.122079+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:29.122079+0000","last_peered":"2024-09-15T03:13:29.122079+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:29.122079+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8757,"log_dups_size":0,"ondisk_log_size":8757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21862002,"num_objects":2669,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8755,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9348","reported_seq":9455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.399491+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:29.399491+0000","last_peered":"2024-09-15T03:13:29.399491+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:29.399491+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23526225,"num_objects":2873,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9347,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":782}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8815","reported_seq":12785,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:30.466519+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:30.466519+0000","last_peered":"2024-09-15T03:13:30.466519+0000","last_clean":"2024-09-15T03:13:30.466519+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:30.466519+0000","last_undegraded":"2024-09-15T03:13:30.466519+0000","last_fullsized":"2024-09-15T03:13:30.466519+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8815,"log_dups_size":0,"ondisk_log_size":8815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041650,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8813,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9239","reported_seq":9345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.260865+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:29.260865+0000","last_peered":"2024-09-15T03:13:29.260865+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:29.260865+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9239,"log_dups_size":0,"ondisk_log_size":9239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23279218,"num_objects":2842,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":758}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9211","reported_seq":9295,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.297528+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:29.297528+0000","last_peered":"2024-09-15T03:13:29.297528+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:29.297528+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9211,"log_dups_size":0,"ondisk_log_size":9211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22831104,"num_objects":2787,"num_object_clones":0,"num_object_copies":11148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":749}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9141","reported_seq":9262,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.216623+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:29.216623+0000","last_peered":"2024-09-15T03:13:29.216623+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:29.216623+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9141,"log_dups_size":0,"ondisk_log_size":9141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23074418,"num_objects":2817,"num_object_clones":0,"num_object_copies":11268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9139,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":791}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9256","reported_seq":9373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.045779+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:29.045779+0000","last_peered":"2024-09-15T03:13:29.045779+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:29.045779+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9256,"log_dups_size":0,"ondisk_log_size":9256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23172722,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9254,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9105","reported_seq":9196,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.241711+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:29.241711+0000","last_peered":"2024-09-15T03:13:29.241711+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:29.241711+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9105,"log_dups_size":0,"ondisk_log_size":9105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23465188,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9102,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9158","reported_seq":9277,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.084293+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:29.084293+0000","last_peered":"2024-09-15T03:13:29.084293+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:29.084293+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9158,"log_dups_size":0,"ondisk_log_size":9158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23017074,"num_objects":2810,"num_object_clones":0,"num_object_copies":11240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":768}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9186","reported_seq":9279,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.336740+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:29.336740+0000","last_peered":"2024-09-15T03:13:29.336740+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:29.336740+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22090155,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9091","reported_seq":13132,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:30.567785+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:30.567785+0000","last_peered":"2024-09-15T03:13:30.567785+0000","last_clean":"2024-09-15T03:13:30.567785+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:30.567785+0000","last_undegraded":"2024-09-15T03:13:30.567785+0000","last_fullsized":"2024-09-15T03:13:30.567785+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23001913,"num_objects":2808,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9225","reported_seq":11209,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.520539+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:29.520539+0000","last_peered":"2024-09-15T03:13:29.520539+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:29.520539+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9225,"log_dups_size":0,"ondisk_log_size":9225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23821113,"num_objects":2908,"num_object_clones":0,"num_object_copies":11632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1819}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8938","reported_seq":9038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.180912+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:29.180912+0000","last_peered":"2024-09-15T03:13:29.180912+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:29.180912+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8938,"log_dups_size":0,"ondisk_log_size":8938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22476402,"num_objects":2744,"num_object_clones":0,"num_object_copies":10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8936,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":731}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9091","reported_seq":9198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.413397+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:29.413397+0000","last_peered":"2024-09-15T03:13:29.413397+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:29.413397+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9091,"log_dups_size":0,"ondisk_log_size":9091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22869618,"num_objects":2792,"num_object_clones":0,"num_object_copies":11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9165","reported_seq":9278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:29.138448+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:29.138448+0000","last_peered":"2024-09-15T03:13:29.138448+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:29.138448+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9165,"log_dups_size":0,"ondisk_log_size":9165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23295602,"num_objects":2844,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9163,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":368179796,"num_objects":44949,"num_object_clones":0,"num_object_copies":179796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44949,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145758,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":735809536,"data_stored":735809536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145785,"ondisk_log_size":145785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491076,"kb_used_data":180732,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93880764,"statfs":{"total":96636764160,"available":96133902336,"internally_reserved":0,"allocated":185069568,"data_stored":184894102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.246}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.311},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.46800000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.495}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482172,"kb_used_data":180660,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93889668,"statfs":{"total":96636764160,"available":96143020032,"internally_reserved":0,"allocated":184995840,"data_stored":184823520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:29 2024","interfaces":[{"interface":"back","average":{"1min":2.637,"5min":2.5680000000000001,"15min":1.512},"min":{"1min":0.184,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":14.653,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.25},{"interface":"front","average":{"1min":1.365,"5min":1.9139999999999999,"15min":1.242},"min":{"1min":0.217,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6980000000000004,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.26900000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":7.4669999999999996},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":7.5119999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.34000000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477472,"kb_used_data":179592,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93894368,"statfs":{"total":96636764160,"available":96147832832,"internally_reserved":0,"allocated":183902208,"data_stored":183726742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,3,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":22.803000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.219}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":27.321999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":2.8620000000000001},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.39800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482748,"kb_used_data":180788,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93889092,"statfs":{"total":96636764160,"available":96142430208,"internally_reserved":0,"allocated":185126912,"data_stored":184951446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,5,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.20300000000000001},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.28999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":0.254},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.40699999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.32700000000000001},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":4.2789999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184233984,"data_stored":184233984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":183410688,"data_stored":183410688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184000512,"data_stored":184000512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":184164352,"data_stored":184164352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:34.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:13:34.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9059", 'reported_seq': 9124, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.293075+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:13:29.293075+0000', 'last_peered': '2024-09-15T03:13:29.293075+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:13:29.293075+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9059, 'log_dups_size': 0, 'ondisk_log_size': 9059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23355392, 'num_objects': 2851, 'num_object_clones': 0, 'num_object_copies': 11404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9059, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 885}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:34.215 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:13:34.215 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8757", 'reported_seq': 8850, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.122079+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:13:29.122079+0000', 'last_peered': '2024-09-15T03:13:29.122079+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:13:29.122079+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8757, 'log_dups_size': 0, 'ondisk_log_size': 8757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21862002, 'num_objects': 2669, 'num_object_clones': 0, 'num_object_copies': 10676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8755, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 685}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9348", 'reported_seq': 9455, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.399491+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:13:29.399491+0000', 'last_peered': '2024-09-15T03:13:29.399491+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:13:29.399491+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9348, 'log_dups_size': 0, 'ondisk_log_size': 9348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23526225, 'num_objects': 2873, 'num_object_clones': 0, 'num_object_copies': 11492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2873, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9347, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 782}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9239", 'reported_seq': 9345, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.260865+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:13:29.260865+0000', 'last_peered': '2024-09-15T03:13:29.260865+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:13:29.260865+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9239, 'log_dups_size': 0, 'ondisk_log_size': 9239, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23279218, 'num_objects': 2842, 'num_object_clones': 0, 'num_object_copies': 11368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9237, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 758}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9211", 'reported_seq': 9295, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.297528+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:13:29.297528+0000', 'last_peered': '2024-09-15T03:13:29.297528+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:13:29.297528+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9211, 'log_dups_size': 0, 'ondisk_log_size': 9211, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22831104, 'num_objects': 2787, 'num_object_clones': 0, 'num_object_copies': 11148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9211, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 749}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9141", 'reported_seq': 9262, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.216623+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:13:29.216623+0000', 'last_peered': '2024-09-15T03:13:29.216623+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:13:29.216623+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9141, 'log_dups_size': 0, 'ondisk_log_size': 9141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23074418, 'num_objects': 2817, 'num_object_clones': 0, 'num_object_copies': 11268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9139, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 791}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9256", 'reported_seq': 9373, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.045779+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:13:29.045779+0000', 'last_peered': '2024-09-15T03:13:29.045779+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:13:29.045779+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9256, 'log_dups_size': 0, 'ondisk_log_size': 9256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23172722, 'num_objects': 2829, 'num_object_clones': 0, 'num_object_copies': 11316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9254, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:34.216 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9105", 'reported_seq': 9196, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.241711+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:13:29.241711+0000', 'last_peered': '2024-09-15T03:13:29.241711+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:13:29.241711+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9105, 'log_dups_size': 0, 'ondisk_log_size': 9105, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23465188, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9102, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 838}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9158", 'reported_seq': 9277, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.084293+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:13:29.084293+0000', 'last_peered': '2024-09-15T03:13:29.084293+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:13:29.084293+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9158, 'log_dups_size': 0, 'ondisk_log_size': 9158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23017074, 'num_objects': 2810, 'num_object_clones': 0, 'num_object_copies': 11240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9156, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 768}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9186", 'reported_seq': 9279, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.336740+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:13:29.336740+0000', 'last_peered': '2024-09-15T03:13:29.336740+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:13:29.336740+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9186, 'log_dups_size': 0, 'ondisk_log_size': 9186, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22090155, 'num_objects': 2697, 'num_object_clones': 0, 'num_object_copies': 10788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9183, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 635}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9225", 'reported_seq': 11209, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.520539+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:13:29.520539+0000', 'last_peered': '2024-09-15T03:13:29.520539+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:13:29.520539+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9225, 'log_dups_size': 0, 'ondisk_log_size': 9225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23821113, 'num_objects': 2908, 'num_object_clones': 0, 'num_object_copies': 11632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1089, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9224, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 952, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1819}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'8938", 'reported_seq': 9038, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.180912+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:13:29.180912+0000', 'last_peered': '2024-09-15T03:13:29.180912+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:13:29.180912+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8938, 'log_dups_size': 0, 'ondisk_log_size': 8938, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22476402, 'num_objects': 2744, 'num_object_clones': 0, 'num_object_copies': 10976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8936, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:13:34.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9091", 'reported_seq': 9198, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.413397+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:13:29.413397+0000', 'last_peered': '2024-09-15T03:13:29.413397+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:13:29.413397+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9091, 'log_dups_size': 0, 'ondisk_log_size': 9091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22869618, 'num_objects': 2792, 'num_object_clones': 0, 'num_object_copies': 11168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9089, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:34.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:13:34.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9165", 'reported_seq': 9278, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:29.138448+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:13:29.138448+0000', 'last_peered': '2024-09-15T03:13:29.138448+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:13:29.138448+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9165, 'log_dups_size': 0, 'ondisk_log_size': 9165, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23295602, 'num_objects': 2844, 'num_object_clones': 0, 'num_object_copies': 11376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9163, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 785}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:34.218 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:13:36.601 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:36.602 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27473/176428 objects degraded (15.572%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":539,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44107,"data_bytes":361282132,"bytes_used":1963257856,"bytes_avail":384583798784,"bytes_total":386547056640,"degraded_objects":27473,"degraded_total":176428,"degraded_ratio":0.15571791325639922,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":232},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-15T03:13:34.867101+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T03:13:36.680 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:13:36.681 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:38.679 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:38.699 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:38.735 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-15T03:13:36.869580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":360274516,"num_objects":43984,"num_object_clones":0,"num_object_copies":175936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43984,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146723,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5211,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146750,"ondisk_log_size":146750,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1917548,"kb_used_data":705836,"kb_used_omap":14,"kb_used_meta":1211633,"kb_avail":375569812,"statfs":{"total":386547056640,"available":384583487488,"internally_reserved":0,"allocated":722776064,"data_stored":722077346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1240712219},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4,13,12,8,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16949248,"num_objects":-2069,"num_object_clones":0,"num_object_copies":-8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2069,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005008"},"pg_stats":[{"pgid":"1.f","version":"35'9118","reported_seq":9184,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.264716+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:34.264716+0000","last_peered":"2024-09-15T03:13:34.264716+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:34.264716+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9118,"log_dups_size":0,"ondisk_log_size":9118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22872064,"num_objects":2792,"num_object_clones":0,"num_object_copies":11168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9118,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8801","reported_seq":8894,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.226745+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:34.226745+0000","last_peered":"2024-09-15T03:13:34.226745+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:34.226745+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8801,"log_dups_size":0,"ondisk_log_size":8801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21501554,"num_objects":2625,"num_object_clones":0,"num_object_copies":10500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8799,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9404","reported_seq":9512,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.576923+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:34.576923+0000","last_peered":"2024-09-15T03:13:34.576923+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:34.576923+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9404,"log_dups_size":0,"ondisk_log_size":9404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23067473,"num_objects":2817,"num_object_clones":0,"num_object_copies":11268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9403,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":726}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8869","reported_seq":12841,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:35.735925+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:35.735925+0000","last_peered":"2024-09-15T03:13:35.735925+0000","last_clean":"2024-09-15T03:13:35.735925+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:35.735925+0000","last_undegraded":"2024-09-15T03:13:35.735925+0000","last_fullsized":"2024-09-15T03:13:35.735925+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8869,"log_dups_size":0,"ondisk_log_size":8869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22599282,"num_objects":2759,"num_object_clones":0,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8867,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9299","reported_seq":9405,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.264645+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:34.264645+0000","last_peered":"2024-09-15T03:13:34.264645+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:34.264645+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9299,"log_dups_size":0,"ondisk_log_size":9299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787698,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9297,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9279","reported_seq":9363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.200477+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:34.200477+0000","last_peered":"2024-09-15T03:13:34.200477+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:34.200477+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22274048,"num_objects":2719,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9279,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":681}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9198","reported_seq":9317,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.256712+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:34.256712+0000","last_peered":"2024-09-15T03:13:34.256712+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:34.256712+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9198,"log_dups_size":0,"ondisk_log_size":9198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22607474,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9196,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":734}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9320","reported_seq":9436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.339172+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:34.339172+0000","last_peered":"2024-09-15T03:13:34.339172+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:34.339172+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9320,"log_dups_size":0,"ondisk_log_size":9320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22648434,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9318,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9175","reported_seq":9266,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.198663+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:34.198663+0000","last_peered":"2024-09-15T03:13:34.198663+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:34.198663+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9175,"log_dups_size":0,"ondisk_log_size":9175,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22891748,"num_objects":2795,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":768}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9217","reported_seq":9335,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.582050+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:34.582050+0000","last_peered":"2024-09-15T03:13:34.582050+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:34.582050+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9217,"log_dups_size":0,"ondisk_log_size":9217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22533746,"num_objects":2751,"num_object_clones":0,"num_object_copies":11004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9215,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9234","reported_seq":9328,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.344336+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:34.344336+0000","last_peered":"2024-09-15T03:13:34.344336+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:34.344336+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21696939,"num_objects":2649,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9231,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":587}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9160","reported_seq":13201,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:35.627631+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:35.627631+0000","last_peered":"2024-09-15T03:13:35.627631+0000","last_clean":"2024-09-15T03:13:35.627631+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:35.627631+0000","last_undegraded":"2024-09-15T03:13:35.627631+0000","last_fullsized":"2024-09-15T03:13:35.627631+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22436665,"num_objects":2739,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9285","reported_seq":11397,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.614917+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:34.614917+0000","last_peered":"2024-09-15T03:13:34.614917+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:34.614917+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9285,"log_dups_size":0,"ondisk_log_size":9285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23329593,"num_objects":2848,"num_object_clones":0,"num_object_copies":11392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1025,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9284,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1016,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'8996","reported_seq":9096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.348635+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:34.348635+0000","last_peered":"2024-09-15T03:13:34.348635+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:34.348635+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8996,"log_dups_size":0,"ondisk_log_size":8996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22001266,"num_objects":2686,"num_object_clones":0,"num_object_copies":10744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8994,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":673}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9162","reported_seq":9269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.362133+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:34.362133+0000","last_peered":"2024-09-15T03:13:34.362133+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:34.362133+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9162,"log_dups_size":0,"ondisk_log_size":9162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22287986,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9160,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":744}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9233","reported_seq":9346,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:34.341733+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:34.341733+0000","last_peered":"2024-09-15T03:13:34.341733+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:34.341733+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9233,"log_dups_size":0,"ondisk_log_size":9233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22738546,"num_objects":2776,"num_object_clones":0,"num_object_copies":11104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9231,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":360274516,"num_objects":43984,"num_object_clones":0,"num_object_copies":175936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43984,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146723,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5211,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":720027648,"data_stored":720027648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146750,"ondisk_log_size":146750,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474792,"kb_used_data":176928,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93897048,"statfs":{"total":96636764160,"available":96150577152,"internally_reserved":0,"allocated":181174272,"data_stored":180998806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,6,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":7.7460000000000004},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":7.6779999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":3.1989999999999998},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":7.7140000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":12.036}]}]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":481860,"kb_used_data":176188,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93889980,"statfs":{"total":96636764160,"available":96143339520,"internally_reserved":0,"allocated":180416512,"data_stored":180244192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.83599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.504},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.53300000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:12:34 2024","interfaces":[{"interface":"back","average":{"1min":3.1240000000000001,"5min":2.2240000000000002,"15min":1.3400000000000001},"min":{"1min":0.29299999999999998,"5min":0.216,"15min":0.216},"max":{"1min":14.709,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.927,"5min":2.3969999999999998,"15min":1.429},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":14.880000000000001,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.91600000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477776,"kb_used_data":175752,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93894064,"statfs":{"total":96636764160,"available":96147521536,"internally_reserved":0,"allocated":179970048,"data_stored":179794582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":15.116},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.50800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":25.184999999999999},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":25.472000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":3.4260000000000002},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.72599999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483120,"kb_used_data":176968,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93888720,"statfs":{"total":96636764160,"available":96142049280,"internally_reserved":0,"allocated":181215232,"data_stored":181039766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,7,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":1.012}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":1.052},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.35099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.317},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.376}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180310016,"data_stored":180310016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179421184,"data_stored":179421184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180060160,"data_stored":180060160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180236288,"data_stored":180236288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:41.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:44.079 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:44.089 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:44.290 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-15T03:13:42.875885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":352361044,"num_objects":43018,"num_object_clones":0,"num_object_copies":172072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147689,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5264,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147716,"ondisk_log_size":147716,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1919220,"kb_used_data":690880,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375568140,"statfs":{"total":386547056640,"available":384581775360,"internally_reserved":0,"allocated":707461120,"data_stored":706762402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,10,9,8,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15818752,"num_objects":-1931,"num_object_clones":0,"num_object_copies":-7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1931,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009834"},"pg_stats":[{"pgid":"1.f","version":"35'9185","reported_seq":9247,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.415565+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:39.415565+0000","last_peered":"2024-09-15T03:13:39.415565+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:39.415565+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9185,"log_dups_size":0,"ondisk_log_size":9185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22323200,"num_objects":2725,"num_object_clones":0,"num_object_copies":10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9185,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8860","reported_seq":8954,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.274975+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:39.274975+0000","last_peered":"2024-09-15T03:13:39.274975+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:39.274975+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8860,"log_dups_size":0,"ondisk_log_size":8860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21018226,"num_objects":2566,"num_object_clones":0,"num_object_copies":10264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8858,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":582}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9454","reported_seq":9562,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.318574+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:39.318574+0000","last_peered":"2024-09-15T03:13:39.318574+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:39.318574+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22657873,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9453,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":676}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8934","reported_seq":12906,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:40.958436+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:40.958436+0000","last_peered":"2024-09-15T03:13:40.958436+0000","last_clean":"2024-09-15T03:13:40.958436+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:40.958436+0000","last_undegraded":"2024-09-15T03:13:40.958436+0000","last_fullsized":"2024-09-15T03:13:40.958436+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8934,"log_dups_size":0,"ondisk_log_size":8934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22066802,"num_objects":2694,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9365","reported_seq":9471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.448444+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:39.448444+0000","last_peered":"2024-09-15T03:13:39.448444+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:39.448444+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9365,"log_dups_size":0,"ondisk_log_size":9365,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22247026,"num_objects":2716,"num_object_clones":0,"num_object_copies":10864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9363,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":632}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9343","reported_seq":9427,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.493758+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:39.493758+0000","last_peered":"2024-09-15T03:13:39.493758+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:39.493758+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9343,"log_dups_size":0,"ondisk_log_size":9343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21749760,"num_objects":2655,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":617}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9254","reported_seq":9375,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.132893+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:39.132893+0000","last_peered":"2024-09-15T03:13:39.132893+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:39.132893+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9254,"log_dups_size":0,"ondisk_log_size":9254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22148722,"num_objects":2704,"num_object_clones":0,"num_object_copies":10816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9252,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":678}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9382","reported_seq":9498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.257975+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:39.257975+0000","last_peered":"2024-09-15T03:13:39.257975+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:39.257975+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22140530,"num_objects":2703,"num_object_clones":0,"num_object_copies":10812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":647}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9237","reported_seq":9328,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.227203+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:39.227203+0000","last_peered":"2024-09-15T03:13:39.227203+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:39.227203+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9237,"log_dups_size":0,"ondisk_log_size":9237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22383844,"num_objects":2733,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":706}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9284","reported_seq":9402,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.527895+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:39.527895+0000","last_peered":"2024-09-15T03:13:39.527895+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:39.527895+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9284,"log_dups_size":0,"ondisk_log_size":9284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21984882,"num_objects":2684,"num_object_clones":0,"num_object_copies":10736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9282,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":642}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9290","reported_seq":9384,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.427284+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:39.427284+0000","last_peered":"2024-09-15T03:13:39.427284+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:39.427284+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21238187,"num_objects":2593,"num_object_clones":0,"num_object_copies":10372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":531}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9223","reported_seq":13264,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:40.705362+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:40.705362+0000","last_peered":"2024-09-15T03:13:40.705362+0000","last_clean":"2024-09-15T03:13:40.705362+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:40.705362+0000","last_undegraded":"2024-09-15T03:13:40.705362+0000","last_fullsized":"2024-09-15T03:13:40.705362+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9223,"log_dups_size":0,"ondisk_log_size":9223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21920569,"num_objects":2676,"num_object_clones":0,"num_object_copies":10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9344","reported_seq":11562,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.652320+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:39.652320+0000","last_peered":"2024-09-15T03:13:39.652320+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:39.652320+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9344,"log_dups_size":0,"ondisk_log_size":9344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22846265,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1069,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9068","reported_seq":9168,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.378216+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:39.378216+0000","last_peered":"2024-09-15T03:13:39.378216+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:39.378216+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9068,"log_dups_size":0,"ondisk_log_size":9068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21411442,"num_objects":2614,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9066,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9205","reported_seq":9312,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.498744+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:39.498744+0000","last_peered":"2024-09-15T03:13:39.498744+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:39.498744+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21935730,"num_objects":2678,"num_object_clones":0,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9203,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":701}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9288","reported_seq":9401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:39.186033+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:39.186033+0000","last_peered":"2024-09-15T03:13:39.186033+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:39.186033+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9288,"log_dups_size":0,"ondisk_log_size":9288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22287986,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9286,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":662}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":352361044,"num_objects":43018,"num_object_clones":0,"num_object_copies":172072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147689,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5264,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":704344064,"data_stored":704344064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147716,"ondisk_log_size":147716,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475196,"kb_used_data":173172,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93896644,"statfs":{"total":96636764160,"available":96150163456,"internally_reserved":0,"allocated":177328128,"data_stored":177152662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,6,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":7.7460000000000004},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":7.6779999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":3.1989999999999998},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":7.7140000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":12.036}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482292,"kb_used_data":172452,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93889548,"statfs":{"total":96636764160,"available":96142897152,"internally_reserved":0,"allocated":176590848,"data_stored":176418528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,5,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":7.3070000000000004},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":5.3949999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.442}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":5.3550000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478164,"kb_used_data":171968,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93893676,"statfs":{"total":96636764160,"available":96147124224,"internally_reserved":0,"allocated":176095232,"data_stored":175919766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,5,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.371}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":1.954}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.41799999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483568,"kb_used_data":173288,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93888272,"statfs":{"total":96636764160,"available":96141590528,"internally_reserved":0,"allocated":177446912,"data_stored":177271446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":6.5499999999999998},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":11.74}]},{"osd":2,"last update":"Sun Sep 15 03:12:39 2024","interfaces":[{"interface":"back","average":{"1min":2.5750000000000002,"5min":3.1949999999999998,"15min":1.8069999999999999},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.16200000000000001},"max":{"1min":20.119,"5min":30.802,"15min":30.802},"last":0.35099999999999998},{"interface":"front","average":{"1min":2.226,"5min":2.052,"15min":1.1879999999999999},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":20.387,"5min":20.387,"15min":20.387},"last":0.622}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.33600000000000002},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176435200,"data_stored":176435200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":175415296,"data_stored":175415296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176185344,"data_stored":176185344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":176308224,"data_stored":176308224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:44.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:46.939 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:47.001 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:47.096 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-15T03:13:46.876956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344570452,"num_objects":42067,"num_object_clones":0,"num_object_copies":168268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42067,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148640,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5332,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148667,"ondisk_log_size":148667,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1899036,"kb_used_data":674832,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375588324,"statfs":{"total":386547056640,"available":384602443776,"internally_reserved":0,"allocated":691027968,"data_stored":690329250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,6,13,15,8,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16711680,"num_objects":-2040,"num_object_clones":0,"num_object_copies":-8160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2040,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009937"},"pg_stats":[{"pgid":"1.f","version":"35'9255","reported_seq":9321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.476487+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:44.476487+0000","last_peered":"2024-09-15T03:13:44.476487+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:44.476487+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9255,"log_dups_size":0,"ondisk_log_size":9255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21749760,"num_objects":2655,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9255,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":689}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8915","reported_seq":9009,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:43.951617+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:43.951617+0000","last_peered":"2024-09-15T03:13:43.951617+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:43.951617+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20567666,"num_objects":2511,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8913,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":527}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9512","reported_seq":9620,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.399449+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:44.399449+0000","last_peered":"2024-09-15T03:13:44.399449+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:44.399449+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9512,"log_dups_size":0,"ondisk_log_size":9512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22182737,"num_objects":2709,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9511,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":618}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8992","reported_seq":12963,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:46.055192+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:46.055192+0000","last_peered":"2024-09-15T03:13:46.055192+0000","last_clean":"2024-09-15T03:13:46.055192+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:46.055192+0000","last_undegraded":"2024-09-15T03:13:46.055192+0000","last_fullsized":"2024-09-15T03:13:46.055192+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21591666,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9423","reported_seq":9529,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.419316+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:44.419316+0000","last_peered":"2024-09-15T03:13:44.419316+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:44.419316+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21771890,"num_objects":2658,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9421,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":574}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9411","reported_seq":9492,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.436623+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:44.436623+0000","last_peered":"2024-09-15T03:13:44.436623+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:44.436623+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9411,"log_dups_size":0,"ondisk_log_size":9411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21192704,"num_objects":2587,"num_object_clones":0,"num_object_copies":10348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9411,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9314","reported_seq":9435,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.332548+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:44.332548+0000","last_peered":"2024-09-15T03:13:44.332548+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:44.332548+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21657202,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":618}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9437","reported_seq":9554,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.216664+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:44.216664+0000","last_peered":"2024-09-15T03:13:44.216664+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:44.216664+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9437,"log_dups_size":0,"ondisk_log_size":9437,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21689970,"num_objects":2648,"num_object_clones":0,"num_object_copies":10592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9435,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":592}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9290","reported_seq":9381,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.396233+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:44.396233+0000","last_peered":"2024-09-15T03:13:44.396233+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:44.396233+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21949668,"num_objects":2680,"num_object_clones":0,"num_object_copies":10720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":653}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9340","reported_seq":9459,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.549033+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:44.549033+0000","last_peered":"2024-09-15T03:13:44.549033+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:44.549033+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21526130,"num_objects":2628,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9338,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":586}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9337","reported_seq":9431,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.444829+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:44.444829+0000","last_peered":"2024-09-15T03:13:44.444829+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:44.444829+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9337,"log_dups_size":0,"ondisk_log_size":9337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20853163,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9334,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":484}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9279","reported_seq":13320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:45.754395+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:45.754395+0000","last_peered":"2024-09-15T03:13:45.754395+0000","last_clean":"2024-09-15T03:13:45.754395+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:45.754395+0000","last_undegraded":"2024-09-15T03:13:45.754395+0000","last_fullsized":"2024-09-15T03:13:45.754395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21461817,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9278,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9405","reported_seq":11758,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.691949+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:44.691949+0000","last_peered":"2024-09-15T03:13:44.691949+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:44.691949+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9405,"log_dups_size":0,"ondisk_log_size":9405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22346553,"num_objects":2728,"num_object_clones":0,"num_object_copies":10912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9404,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1137,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1824}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9132","reported_seq":9232,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.518563+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:44.518563+0000","last_peered":"2024-09-15T03:13:44.518563+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:44.518563+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9132,"log_dups_size":0,"ondisk_log_size":9132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20887154,"num_objects":2550,"num_object_clones":0,"num_object_copies":10200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":537}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9269","reported_seq":9375,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.527830+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:44.527830+0000","last_peered":"2024-09-15T03:13:44.527830+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:44.527830+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9269,"log_dups_size":0,"ondisk_log_size":9269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21411442,"num_objects":2614,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9356","reported_seq":9469,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:44.195068+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:44.195068+0000","last_peered":"2024-09-15T03:13:44.195068+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:44.195068+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9356,"log_dups_size":0,"ondisk_log_size":9356,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730930,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9354,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":594}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":344570452,"num_objects":42067,"num_object_clones":0,"num_object_copies":168268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42067,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148640,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5332,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":688484352,"data_stored":688484352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148667,"ondisk_log_size":148667,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479660,"kb_used_data":169288,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93892180,"statfs":{"total":96636764160,"available":96145592320,"internally_reserved":0,"allocated":173350912,"data_stored":173175446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,6,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":1.54,"15min":1.696},"min":{"1min":0.183,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":5.0780000000000003,"5min":13.266999999999999,"15min":17.408999999999999},"last":3.6509999999999998},{"interface":"front","average":{"1min":1.242,"5min":1.9710000000000001,"15min":2.0190000000000001},"min":{"1min":0.246,"5min":0.185,"15min":0.185},"max":{"1min":8.2449999999999992,"5min":26.143000000000001,"15min":26.143000000000001},"last":3.839}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":3.6829999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:12:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":3.1619999999999999,"15min":1.5229999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.20200000000000001},"max":{"1min":8.5830000000000002,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.2829999999999999,"5min":2.3420000000000001,"15min":1.4350000000000001},"min":{"1min":0.24099999999999999,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":5.0359999999999996,"5min":32.377000000000002,"15min":32.377000000000002},"last":3.7200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466012,"kb_used_data":168656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93905828,"statfs":{"total":96636764160,"available":96159567872,"internally_reserved":0,"allocated":172703744,"data_stored":172531424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,2,3,0,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.20499999999999999},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":3.9990000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.9470000000000001},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":4.1900000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":4.226},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":4.0990000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482648,"kb_used_data":168132,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93889192,"statfs":{"total":96636764160,"available":96142532608,"internally_reserved":0,"allocated":172167168,"data_stored":171991702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,5,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.879},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.245}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":0.83999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.63}]}]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470716,"kb_used_data":168756,"kb_used_omap":4,"kb_used_meta":301947,"kb_avail":93901124,"statfs":{"total":96636764160,"available":96154750976,"internally_reserved":0,"allocated":172806144,"data_stored":172630678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309194305},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.625},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.65900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.40500000000000003},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.26500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:12:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.7430000000000001,"15min":1.171},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.199000000000002,"5min":20.948,"15min":20.948},"last":0.29999999999999999},{"interface":"front","average":{"1min":3.6619999999999999,"5min":2.194,"15min":1.131},"min":{"1min":0.29299999999999998,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":30.013000000000002,"5min":30.013000000000002,"15min":30.013000000000002},"last":5.6509999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172494848,"data_stored":172494848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171368448,"data_stored":171368448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172253184,"data_stored":172253184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172367872,"data_stored":172367872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:47.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:13:47.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9255", 'reported_seq': 9321, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.476487+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:13:44.476487+0000', 'last_peered': '2024-09-15T03:13:44.476487+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:13:44.476487+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9255, 'log_dups_size': 0, 'ondisk_log_size': 9255, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21749760, 'num_objects': 2655, 'num_object_clones': 0, 'num_object_copies': 10620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9255, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 689}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:47.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:13:47.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'8915", 'reported_seq': 9009, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:43.951617+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:13:43.951617+0000', 'last_peered': '2024-09-15T03:13:43.951617+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:13:43.951617+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 8915, 'log_dups_size': 0, 'ondisk_log_size': 8915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20567666, 'num_objects': 2511, 'num_object_clones': 0, 'num_object_copies': 10044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8913, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 527}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:47.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:13:47.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9512", 'reported_seq': 9620, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.399449+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:13:44.399449+0000', 'last_peered': '2024-09-15T03:13:44.399449+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:13:44.399449+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9512, 'log_dups_size': 0, 'ondisk_log_size': 9512, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22182737, 'num_objects': 2709, 'num_object_clones': 0, 'num_object_copies': 10836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2709, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9511, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 618}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:47.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:13:47.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9423", 'reported_seq': 9529, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.419316+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:13:44.419316+0000', 'last_peered': '2024-09-15T03:13:44.419316+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:13:44.419316+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9423, 'log_dups_size': 0, 'ondisk_log_size': 9423, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21771890, 'num_objects': 2658, 'num_object_clones': 0, 'num_object_copies': 10632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9421, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 574}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9411", 'reported_seq': 9492, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.436623+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:13:44.436623+0000', 'last_peered': '2024-09-15T03:13:44.436623+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:13:44.436623+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9411, 'log_dups_size': 0, 'ondisk_log_size': 9411, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21192704, 'num_objects': 2587, 'num_object_clones': 0, 'num_object_copies': 10348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9411, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 549}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9314", 'reported_seq': 9435, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.332548+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:13:44.332548+0000', 'last_peered': '2024-09-15T03:13:44.332548+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:13:44.332548+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9314, 'log_dups_size': 0, 'ondisk_log_size': 9314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21657202, 'num_objects': 2644, 'num_object_clones': 0, 'num_object_copies': 10576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9312, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 618}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9437", 'reported_seq': 9554, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.216664+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:13:44.216664+0000', 'last_peered': '2024-09-15T03:13:44.216664+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:13:44.216664+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9437, 'log_dups_size': 0, 'ondisk_log_size': 9437, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21689970, 'num_objects': 2648, 'num_object_clones': 0, 'num_object_copies': 10592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9435, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9290", 'reported_seq': 9381, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.396233+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:13:44.396233+0000', 'last_peered': '2024-09-15T03:13:44.396233+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:13:44.396233+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9290, 'log_dups_size': 0, 'ondisk_log_size': 9290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21949668, 'num_objects': 2680, 'num_object_clones': 0, 'num_object_copies': 10720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9287, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 653}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9340", 'reported_seq': 9459, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.549033+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:13:44.549033+0000', 'last_peered': '2024-09-15T03:13:44.549033+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:13:44.549033+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9340, 'log_dups_size': 0, 'ondisk_log_size': 9340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21526130, 'num_objects': 2628, 'num_object_clones': 0, 'num_object_copies': 10512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9338, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 586}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:13:47.099 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9337", 'reported_seq': 9431, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.444829+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:13:44.444829+0000', 'last_peered': '2024-09-15T03:13:44.444829+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:13:44.444829+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9337, 'log_dups_size': 0, 'ondisk_log_size': 9337, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20853163, 'num_objects': 2546, 'num_object_clones': 0, 'num_object_copies': 10184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9334, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 484}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:13:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9405", 'reported_seq': 11758, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.691949+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:13:44.691949+0000', 'last_peered': '2024-09-15T03:13:44.691949+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:13:44.691949+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9405, 'log_dups_size': 0, 'ondisk_log_size': 9405, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22346553, 'num_objects': 2728, 'num_object_clones': 0, 'num_object_copies': 10912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 904, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9404, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1137, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:13:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9132", 'reported_seq': 9232, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.518563+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:13:44.518563+0000', 'last_peered': '2024-09-15T03:13:44.518563+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:13:44.518563+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9132, 'log_dups_size': 0, 'ondisk_log_size': 9132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20887154, 'num_objects': 2550, 'num_object_clones': 0, 'num_object_copies': 10200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9130, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 537}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:13:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9269", 'reported_seq': 9375, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.527830+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:13:44.527830+0000', 'last_peered': '2024-09-15T03:13:44.527830+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:13:44.527830+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9269, 'log_dups_size': 0, 'ondisk_log_size': 9269, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21411442, 'num_objects': 2614, 'num_object_clones': 0, 'num_object_copies': 10456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9267, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 637}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:13:47.100 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9356", 'reported_seq': 9469, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:44.195068+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:13:44.195068+0000', 'last_peered': '2024-09-15T03:13:44.195068+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:13:44.195068+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9356, 'log_dups_size': 0, 'ondisk_log_size': 9356, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21730930, 'num_objects': 2653, 'num_object_clones': 0, 'num_object_copies': 10612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9354, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 594}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:47.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:13:49.315 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:49.315 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27352/168268 objects degraded (16.255%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":552,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42067,"data_bytes":344570452,"bytes_used":1944612864,"bytes_avail":384602443776,"bytes_total":386547056640,"degraded_objects":27352,"degraded_total":168268,"degraded_ratio":0.16255021751016238,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":159},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:13:49.341 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:13:49.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:51.062 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:51.104 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:51.139 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-15T03:13:50.877892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":337795668,"num_objects":41240,"num_object_clones":0,"num_object_copies":164960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41240,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149467,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5384,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149494,"ondisk_log_size":149494,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1900408,"kb_used_data":663740,"kb_used_omap":14,"kb_used_meta":1236593,"kb_avail":375586952,"statfs":{"total":386547056640,"available":384601038848,"internally_reserved":0,"allocated":679669760,"data_stored":678971042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1266271259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,7,16,8,15,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22478848,"num_objects":-2744,"num_object_clones":0,"num_object_copies":-10976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2744,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007784"},"pg_stats":[{"pgid":"1.f","version":"35'9311","reported_seq":9377,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.420502+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:49.420502+0000","last_peered":"2024-09-15T03:13:49.420502+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:49.420502+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9311,"log_dups_size":0,"ondisk_log_size":9311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21291008,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9311,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'8962","reported_seq":9053,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.302757+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:49.302757+0000","last_peered":"2024-09-15T03:13:49.302757+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:49.302757+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20182642,"num_objects":2464,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8960,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":480}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9572","reported_seq":9680,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.491562+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:49.491562+0000","last_peered":"2024-09-15T03:13:49.491562+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:49.491562+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9572,"log_dups_size":0,"ondisk_log_size":9572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21691217,"num_objects":2649,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2649,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9571,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":558}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'8992","reported_seq":12963,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:46.055192+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:46.055192+0000","last_peered":"2024-09-15T03:13:46.055192+0000","last_clean":"2024-09-15T03:13:46.055192+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:46.055192+0000","last_undegraded":"2024-09-15T03:13:46.055192+0000","last_fullsized":"2024-09-15T03:13:46.055192+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":8992,"log_dups_size":0,"ondisk_log_size":8992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21591666,"num_objects":2636,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9477","reported_seq":9583,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.524864+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:49.524864+0000","last_peered":"2024-09-15T03:13:49.524864+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:49.524864+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9477,"log_dups_size":0,"ondisk_log_size":9477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21329522,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9475,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9474","reported_seq":9558,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.407606+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:49.407606+0000","last_peered":"2024-09-15T03:13:49.407606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:49.407606+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9474,"log_dups_size":0,"ondisk_log_size":9474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20676608,"num_objects":2524,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9379","reported_seq":9500,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.354326+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:49.354326+0000","last_peered":"2024-09-15T03:13:49.354326+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:49.354326+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9379,"log_dups_size":0,"ondisk_log_size":9379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21124722,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9377,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9501","reported_seq":9617,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.271016+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:49.271016+0000","last_peered":"2024-09-15T03:13:49.271016+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:49.271016+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9501,"log_dups_size":0,"ondisk_log_size":9501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21165682,"num_objects":2584,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":528}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9359","reported_seq":9450,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.223750+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:49.223750+0000","last_peered":"2024-09-15T03:13:49.223750+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:49.223750+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21384420,"num_objects":2611,"num_object_clones":0,"num_object_copies":10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9356,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":584}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9384","reported_seq":9503,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.644369+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:49.644369+0000","last_peered":"2024-09-15T03:13:49.644369+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:49.644369+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9384,"log_dups_size":0,"ondisk_log_size":9384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21165682,"num_objects":2584,"num_object_clones":0,"num_object_copies":10336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9382,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":542}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9386","reported_seq":9480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.420457+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:49.420457+0000","last_peered":"2024-09-15T03:13:49.420457+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:49.420457+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9386,"log_dups_size":0,"ondisk_log_size":9386,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20451755,"num_objects":2497,"num_object_clones":0,"num_object_copies":9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9383,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":435}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9279","reported_seq":13320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:45.754395+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:45.754395+0000","last_peered":"2024-09-15T03:13:45.754395+0000","last_clean":"2024-09-15T03:13:45.754395+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:45.754395+0000","last_undegraded":"2024-09-15T03:13:45.754395+0000","last_fullsized":"2024-09-15T03:13:45.754395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9279,"log_dups_size":0,"ondisk_log_size":9279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21461817,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9278,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9475","reported_seq":11933,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.676311+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:49.676311+0000","last_peered":"2024-09-15T03:13:49.676311+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:49.676311+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9475,"log_dups_size":0,"ondisk_log_size":9475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21773113,"num_objects":2658,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":852,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1189,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9196","reported_seq":9296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.584691+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:49.584691+0000","last_peered":"2024-09-15T03:13:49.584691+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:49.584691+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9196,"log_dups_size":0,"ondisk_log_size":9196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20362866,"num_objects":2486,"num_object_clones":0,"num_object_copies":9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9194,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":473}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9318","reported_seq":9425,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.348231+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:49.348231+0000","last_peered":"2024-09-15T03:13:49.348231+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:49.348231+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9318,"log_dups_size":0,"ondisk_log_size":9318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21010034,"num_objects":2565,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9316,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9429","reported_seq":9541,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:49.293145+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:49.293145+0000","last_peered":"2024-09-15T03:13:49.293145+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:49.293145+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9429,"log_dups_size":0,"ondisk_log_size":9429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21132914,"num_objects":2580,"num_object_clones":0,"num_object_copies":10320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9427,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":521}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":337795668,"num_objects":41240,"num_object_clones":0,"num_object_copies":164960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41240,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149467,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5384,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":677003264,"data_stored":677003264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149494,"ondisk_log_size":149494,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480088,"kb_used_data":165584,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93891752,"statfs":{"total":96636764160,"available":96145154048,"internally_reserved":0,"allocated":169558016,"data_stored":169382550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,4,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":5.0039999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.30499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.34000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466504,"kb_used_data":164992,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93905336,"statfs":{"total":96636764160,"available":96159064064,"internally_reserved":0,"allocated":168951808,"data_stored":168779488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,6,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.26300000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.7050000000000001},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.7389999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.30199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482648,"kb_used_data":168132,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93889192,"statfs":{"total":96636764160,"available":96142532608,"internally_reserved":0,"allocated":172167168,"data_stored":171991702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,3,5,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.879},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.245}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":0.83999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.63}]}]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471168,"kb_used_data":165032,"kb_used_omap":4,"kb_used_meta":306107,"kb_avail":93900672,"statfs":{"total":96636764160,"available":96154288128,"internally_reserved":0,"allocated":168992768,"data_stored":168817302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313454145},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.496},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.46100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.42399999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.56499999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":14.339},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":14.422000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":168681472,"data_stored":168681472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171368448,"data_stored":171368448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":168443904,"data_stored":168443904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":168509440,"data_stored":168509440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:54.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:55.867 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:55.937 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:56.013 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-15T03:13:54.879877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":330070612,"num_objects":40297,"num_object_clones":0,"num_object_copies":161188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150410,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5447,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150437,"ondisk_log_size":150437,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1872248,"kb_used_data":648080,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375615112,"statfs":{"total":386547056640,"available":384629874688,"internally_reserved":0,"allocated":663633920,"data_stored":662935202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,9,11,12,21,18,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22290432,"num_objects":-2721,"num_object_clones":0,"num_object_copies":-10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2721,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003992"},"pg_stats":[{"pgid":"1.f","version":"35'9352","reported_seq":9418,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.388669+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:54.388669+0000","last_peered":"2024-09-15T03:13:54.388669+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:54.388669+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20955136,"num_objects":2558,"num_object_clones":0,"num_object_copies":10232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9022","reported_seq":9114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.087805+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:54.087805+0000","last_peered":"2024-09-15T03:13:54.087805+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:54.087805+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19691122,"num_objects":2404,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9622","reported_seq":9730,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.578513+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:54.578513+0000","last_peered":"2024-09-15T03:13:54.578513+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:54.578513+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21281617,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9621,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":508}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9036","reported_seq":13008,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:51.020608+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:51.020608+0000","last_peered":"2024-09-15T03:13:51.020608+0000","last_clean":"2024-09-15T03:13:51.020608+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:51.020608+0000","last_undegraded":"2024-09-15T03:13:51.020608+0000","last_fullsized":"2024-09-15T03:13:51.020608+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9036,"log_dups_size":0,"ondisk_log_size":9036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21231218,"num_objects":2592,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9034,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9526","reported_seq":9632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.349012+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:54.349012+0000","last_peered":"2024-09-15T03:13:54.349012+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:54.349012+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20928114,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9524,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":471}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9544","reported_seq":9628,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.540792+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:54.540792+0000","last_peered":"2024-09-15T03:13:54.540792+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:54.540792+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20103168,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9444","reported_seq":9565,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.214155+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:54.214155+0000","last_peered":"2024-09-15T03:13:54.214155+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:54.214155+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9444,"log_dups_size":0,"ondisk_log_size":9444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20592242,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9442,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":488}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9561","reported_seq":9678,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.371261+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:54.371261+0000","last_peered":"2024-09-15T03:13:54.371261+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:54.371261+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9561,"log_dups_size":0,"ondisk_log_size":9561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20674162,"num_objects":2524,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9559,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9419","reported_seq":9510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.540754+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:54.540754+0000","last_peered":"2024-09-15T03:13:54.540754+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:54.540754+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9419,"log_dups_size":0,"ondisk_log_size":9419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20892900,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9447","reported_seq":9566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.637965+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:54.637965+0000","last_peered":"2024-09-15T03:13:54.637965+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:54.637965+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9447,"log_dups_size":0,"ondisk_log_size":9447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649586,"num_objects":2521,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9445,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":479}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9458","reported_seq":9552,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.629975+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:54.629975+0000","last_peered":"2024-09-15T03:13:54.629975+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:54.629975+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9458,"log_dups_size":0,"ondisk_log_size":9458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19861931,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9455,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9354","reported_seq":13395,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:51.122133+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:51.122133+0000","last_peered":"2024-09-15T03:13:51.122133+0000","last_clean":"2024-09-15T03:13:51.122133+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:51.122133+0000","last_undegraded":"2024-09-15T03:13:51.122133+0000","last_fullsized":"2024-09-15T03:13:51.122133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20847417,"num_objects":2545,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9526","reported_seq":12110,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.704090+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:54.704090+0000","last_peered":"2024-09-15T03:13:54.704090+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:54.704090+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21355321,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9525,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9251","reported_seq":9351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.389567+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:54.389567+0000","last_peered":"2024-09-15T03:13:54.389567+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:54.389567+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19912306,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":418}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9383","reported_seq":9487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.644344+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:54.644344+0000","last_peered":"2024-09-15T03:13:54.644344+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:54.644344+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9383,"log_dups_size":0,"ondisk_log_size":9383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20477554,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9381,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9492","reported_seq":9605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.336235+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:54.336235+0000","last_peered":"2024-09-15T03:13:54.336235+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:54.336235+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9492,"log_dups_size":0,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20616818,"num_objects":2517,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":458}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":330070612,"num_objects":40297,"num_object_clones":0,"num_object_copies":161188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40297,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150410,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5447,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":661430272,"data_stored":661430272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150437,"ondisk_log_size":150437,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463160,"kb_used_data":161136,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93908680,"statfs":{"total":96636764160,"available":96162488320,"internally_reserved":0,"allocated":165003264,"data_stored":164827798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,0,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.44},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.26700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":1.306},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":1.3240000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.47699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":161188,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":165056512,"data_stored":164884192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":1.117},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.64500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.56999999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466508,"kb_used_data":164476,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93905332,"statfs":{"total":96636764160,"available":96159059968,"internally_reserved":0,"allocated":168423424,"data_stored":168247958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,6,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":5.9939999999999998},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.26800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.218},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":6.0380000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":1.8280000000000001},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471560,"kb_used_data":161280,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93900280,"statfs":{"total":96636764160,"available":96153886720,"internally_reserved":0,"allocated":165150720,"data_stored":164975254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,1,1,3,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":4.4450000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":4.7859999999999996},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.42699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.39000000000000001},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164777984,"data_stored":164777984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167469056,"data_stored":167469056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164573184,"data_stored":164573184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":164610048,"data_stored":164610048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:56.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:13:58.019 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:13:58.030 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:13:58.046 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-15T03:13:56.880410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329087572,"num_objects":40177,"num_object_clones":0,"num_object_copies":160708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150530,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5447,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150557,"ondisk_log_size":150557,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1871888,"kb_used_data":643552,"kb_used_omap":14,"kb_used_meta":1228273,"kb_avail":375615472,"statfs":{"total":386547056640,"available":384630243328,"internally_reserved":0,"allocated":658997248,"data_stored":658298530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1257751579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,8,8,9,14,17,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16416768,"num_objects":-2004,"num_object_clones":0,"num_object_copies":-8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003999"},"pg_stats":[{"pgid":"1.f","version":"35'9352","reported_seq":9418,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.388669+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:54.388669+0000","last_peered":"2024-09-15T03:13:54.388669+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:54.388669+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20955136,"num_objects":2558,"num_object_clones":0,"num_object_copies":10232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9022","reported_seq":9114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.087805+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:54.087805+0000","last_peered":"2024-09-15T03:13:54.087805+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:54.087805+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19691122,"num_objects":2404,"num_object_clones":0,"num_object_copies":9616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":420}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9622","reported_seq":9730,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.578513+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:54.578513+0000","last_peered":"2024-09-15T03:13:54.578513+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:54.578513+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9622,"log_dups_size":0,"ondisk_log_size":9622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21281617,"num_objects":2599,"num_object_clones":0,"num_object_copies":10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9621,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":508}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9097","reported_seq":13069,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:56.475984+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:56.475984+0000","last_peered":"2024-09-15T03:13:56.475984+0000","last_clean":"2024-09-15T03:13:56.475984+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:56.475984+0000","last_undegraded":"2024-09-15T03:13:56.475984+0000","last_fullsized":"2024-09-15T03:13:56.475984+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20731506,"num_objects":2531,"num_object_clones":0,"num_object_copies":10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9526","reported_seq":9632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.349012+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:54.349012+0000","last_peered":"2024-09-15T03:13:54.349012+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:54.349012+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20928114,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9524,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":471}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9544","reported_seq":9628,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.540792+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:54.540792+0000","last_peered":"2024-09-15T03:13:54.540792+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:54.540792+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9544,"log_dups_size":0,"ondisk_log_size":9544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20103168,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9544,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":416}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9444","reported_seq":9565,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.214155+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:54.214155+0000","last_peered":"2024-09-15T03:13:54.214155+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:54.214155+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9444,"log_dups_size":0,"ondisk_log_size":9444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20592242,"num_objects":2514,"num_object_clones":0,"num_object_copies":10056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9442,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":488}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9561","reported_seq":9678,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.371261+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:54.371261+0000","last_peered":"2024-09-15T03:13:54.371261+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:54.371261+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9561,"log_dups_size":0,"ondisk_log_size":9561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20674162,"num_objects":2524,"num_object_clones":0,"num_object_copies":10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9559,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9419","reported_seq":9510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.540754+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:54.540754+0000","last_peered":"2024-09-15T03:13:54.540754+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:54.540754+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9419,"log_dups_size":0,"ondisk_log_size":9419,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20892900,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9447","reported_seq":9566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.637965+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:54.637965+0000","last_peered":"2024-09-15T03:13:54.637965+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:54.637965+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9447,"log_dups_size":0,"ondisk_log_size":9447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20649586,"num_objects":2521,"num_object_clones":0,"num_object_copies":10084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9445,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":479}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9458","reported_seq":9552,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.629975+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:54.629975+0000","last_peered":"2024-09-15T03:13:54.629975+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:54.629975+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9458,"log_dups_size":0,"ondisk_log_size":9458,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19861931,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9455,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9413","reported_seq":13454,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:56.088799+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:56.088799+0000","last_peered":"2024-09-15T03:13:56.088799+0000","last_clean":"2024-09-15T03:13:56.088799+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:56.088799+0000","last_undegraded":"2024-09-15T03:13:56.088799+0000","last_fullsized":"2024-09-15T03:13:56.088799+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9413,"log_dups_size":0,"ondisk_log_size":9413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20364089,"num_objects":2486,"num_object_clones":0,"num_object_copies":9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9526","reported_seq":12110,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.704090+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:54.704090+0000","last_peered":"2024-09-15T03:13:54.704090+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:54.704090+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9526,"log_dups_size":0,"ondisk_log_size":9526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21355321,"num_objects":2607,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":789,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9525,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9251","reported_seq":9351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.389567+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:54.389567+0000","last_peered":"2024-09-15T03:13:54.389567+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:54.389567+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19912306,"num_objects":2431,"num_object_clones":0,"num_object_copies":9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":418}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9383","reported_seq":9487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.644344+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:54.644344+0000","last_peered":"2024-09-15T03:13:54.644344+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:54.644344+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9383,"log_dups_size":0,"ondisk_log_size":9383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20477554,"num_objects":2500,"num_object_clones":0,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9381,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9492","reported_seq":9605,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:54.336235+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:54.336235+0000","last_peered":"2024-09-15T03:13:54.336235+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:54.336235+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9492,"log_dups_size":0,"ondisk_log_size":9492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20616818,"num_objects":2517,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":458}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":329087572,"num_objects":40177,"num_object_clones":0,"num_object_copies":160708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40177,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150530,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5447,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":657399808,"data_stored":657399808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150557,"ondisk_log_size":150557,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463160,"kb_used_data":161136,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93908680,"statfs":{"total":96636764160,"available":96162488320,"internally_reserved":0,"allocated":165003264,"data_stored":164827798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,0,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.44},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.26700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":1.306},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":1.3240000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.47699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":161188,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":165056512,"data_stored":164884192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":1.117},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.64500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.56999999999999995}]}]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466148,"kb_used_data":159948,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93905692,"statfs":{"total":96636764160,"available":96159428608,"internally_reserved":0,"allocated":163786752,"data_stored":163611286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":28.076000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.32800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":28.145}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":1.5580000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471560,"kb_used_data":161280,"kb_used_omap":4,"kb_used_meta":310267,"kb_avail":93900280,"statfs":{"total":96636764160,"available":96153886720,"internally_reserved":0,"allocated":165150720,"data_stored":164975254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317713985},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,1,1,3,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":4.4450000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":4.7859999999999996},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.42699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.39000000000000001},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164777984,"data_stored":164777984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163438592,"data_stored":163438592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164573184,"data_stored":164573184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":164610048,"data_stored":164610048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:13:58.047 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:13:58.047 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9352", 'reported_seq': 9418, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.388669+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:13:54.388669+0000', 'last_peered': '2024-09-15T03:13:54.388669+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:13:54.388669+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9352, 'log_dups_size': 0, 'ondisk_log_size': 9352, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20955136, 'num_objects': 2558, 'num_object_clones': 0, 'num_object_copies': 10232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9352, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 592}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9022", 'reported_seq': 9114, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.087805+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:13:54.087805+0000', 'last_peered': '2024-09-15T03:13:54.087805+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:13:54.087805+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9022, 'log_dups_size': 0, 'ondisk_log_size': 9022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19691122, 'num_objects': 2404, 'num_object_clones': 0, 'num_object_copies': 9616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9020, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9622", 'reported_seq': 9730, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.578513+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:13:54.578513+0000', 'last_peered': '2024-09-15T03:13:54.578513+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:13:54.578513+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9622, 'log_dups_size': 0, 'ondisk_log_size': 9622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21281617, 'num_objects': 2599, 'num_object_clones': 0, 'num_object_copies': 10396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2599, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9621, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 508}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9526", 'reported_seq': 9632, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.349012+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:13:54.349012+0000', 'last_peered': '2024-09-15T03:13:54.349012+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:13:54.349012+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9526, 'log_dups_size': 0, 'ondisk_log_size': 9526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20928114, 'num_objects': 2555, 'num_object_clones': 0, 'num_object_copies': 10220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9524, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 471}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9544", 'reported_seq': 9628, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.540792+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:13:54.540792+0000', 'last_peered': '2024-09-15T03:13:54.540792+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:13:54.540792+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9544, 'log_dups_size': 0, 'ondisk_log_size': 9544, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20103168, 'num_objects': 2454, 'num_object_clones': 0, 'num_object_copies': 9816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9544, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 416}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9444", 'reported_seq': 9565, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.214155+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:13:54.214155+0000', 'last_peered': '2024-09-15T03:13:54.214155+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:13:54.214155+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9444, 'log_dups_size': 0, 'ondisk_log_size': 9444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20592242, 'num_objects': 2514, 'num_object_clones': 0, 'num_object_copies': 10056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9442, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 488}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:58.048 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9561", 'reported_seq': 9678, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.371261+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:13:54.371261+0000', 'last_peered': '2024-09-15T03:13:54.371261+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:13:54.371261+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9561, 'log_dups_size': 0, 'ondisk_log_size': 9561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20674162, 'num_objects': 2524, 'num_object_clones': 0, 'num_object_copies': 10096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9559, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 468}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9419", 'reported_seq': 9510, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.540754+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:13:54.540754+0000', 'last_peered': '2024-09-15T03:13:54.540754+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:13:54.540754+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9419, 'log_dups_size': 0, 'ondisk_log_size': 9419, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20892900, 'num_objects': 2551, 'num_object_clones': 0, 'num_object_copies': 10204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2551, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9416, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 524}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9447", 'reported_seq': 9566, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.637965+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:13:54.637965+0000', 'last_peered': '2024-09-15T03:13:54.637965+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:13:54.637965+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9447, 'log_dups_size': 0, 'ondisk_log_size': 9447, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20649586, 'num_objects': 2521, 'num_object_clones': 0, 'num_object_copies': 10084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9445, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 479}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9458", 'reported_seq': 9552, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.629975+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:13:54.629975+0000', 'last_peered': '2024-09-15T03:13:54.629975+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:13:54.629975+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9458, 'log_dups_size': 0, 'ondisk_log_size': 9458, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19861931, 'num_objects': 2425, 'num_object_clones': 0, 'num_object_copies': 9700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9455, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 363}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9526", 'reported_seq': 12110, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.704090+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:13:54.704090+0000', 'last_peered': '2024-09-15T03:13:54.704090+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:13:54.704090+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9526, 'log_dups_size': 0, 'ondisk_log_size': 9526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21355321, 'num_objects': 2607, 'num_object_clones': 0, 'num_object_copies': 10428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 789, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9525, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1252, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:58.049 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:13:58.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9251", 'reported_seq': 9351, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.389567+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:13:54.389567+0000', 'last_peered': '2024-09-15T03:13:54.389567+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:13:54.389567+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9251, 'log_dups_size': 0, 'ondisk_log_size': 9251, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19912306, 'num_objects': 2431, 'num_object_clones': 0, 'num_object_copies': 9724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9249, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:13:58.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:13:58.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9383", 'reported_seq': 9487, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.644344+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:13:54.644344+0000', 'last_peered': '2024-09-15T03:13:54.644344+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:13:54.644344+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9383, 'log_dups_size': 0, 'ondisk_log_size': 9383, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20477554, 'num_objects': 2500, 'num_object_clones': 0, 'num_object_copies': 10000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9381, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:13:58.050 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:13:58.050 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9492", 'reported_seq': 9605, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:13:54.336235+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:13:54.336235+0000', 'last_peered': '2024-09-15T03:13:54.336235+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:13:54.336235+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9492, 'log_dups_size': 0, 'ondisk_log_size': 9492, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20616818, 'num_objects': 2517, 'num_object_clones': 0, 'num_object_copies': 10068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9490, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 458}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:13:58.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:14:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:00.083 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27237/160708 objects degraded (16.948%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":563,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40177,"data_bytes":329087572,"bytes_used":1916813312,"bytes_avail":384630243328,"bytes_total":386547056640,"degraded_objects":27237,"degraded_total":160708,"degraded_ratio":0.16948129526843714,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":115,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":157},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:14:00.227 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:14:00.227 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:01.942 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:01.987 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:02.077 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-15T03:14:00.881334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":322329172,"num_objects":39352,"num_object_clones":0,"num_object_copies":157408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151355,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5501,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151382,"ondisk_log_size":151382,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1856532,"kb_used_data":632356,"kb_used_omap":14,"kb_used_meta":1224113,"kb_avail":375630828,"statfs":{"total":386547056640,"available":384645967872,"internally_reserved":0,"allocated":647532544,"data_stored":646833826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":1253491739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,3,6,2,16,12,15,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22241280,"num_objects":-2715,"num_object_clones":0,"num_object_copies":-10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2715,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003919"},"pg_stats":[{"pgid":"1.f","version":"35'9403","reported_seq":9469,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.612530+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:13:59.612530+0000","last_peered":"2024-09-15T03:13:59.612530+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:13:59.612530+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9403,"log_dups_size":0,"ondisk_log_size":9403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20537344,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9403,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9083","reported_seq":9177,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.394669+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:13:59.394669+0000","last_peered":"2024-09-15T03:13:59.394669+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:13:59.394669+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9083,"log_dups_size":0,"ondisk_log_size":9083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19191410,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9679","reported_seq":9787,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.645754+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:13:59.645754+0000","last_peered":"2024-09-15T03:13:59.645754+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:13:59.645754+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9679,"log_dups_size":0,"ondisk_log_size":9679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20814673,"num_objects":2542,"num_object_clones":0,"num_object_copies":10168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9678,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":451}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9097","reported_seq":13069,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:56.475984+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:13:56.475984+0000","last_peered":"2024-09-15T03:13:56.475984+0000","last_clean":"2024-09-15T03:13:56.475984+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:13:56.475984+0000","last_undegraded":"2024-09-15T03:13:56.475984+0000","last_fullsized":"2024-09-15T03:13:56.475984+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20731506,"num_objects":2531,"num_object_clones":0,"num_object_copies":10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9594","reported_seq":9700,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.645281+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:13:59.645281+0000","last_peered":"2024-09-15T03:13:59.645281+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:13:59.645281+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9594,"log_dups_size":0,"ondisk_log_size":9594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20371058,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":403}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9603","reported_seq":9687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.671409+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:13:59.671409+0000","last_peered":"2024-09-15T03:13:59.671409+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:13:59.671409+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19619840,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9603,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9502","reported_seq":9622,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.428615+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:13:59.428615+0000","last_peered":"2024-09-15T03:13:59.428615+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:13:59.428615+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20117106,"num_objects":2456,"num_object_clones":0,"num_object_copies":9824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":430}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9612","reported_seq":9729,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.289279+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:13:59.289279+0000","last_peered":"2024-09-15T03:13:59.289279+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:13:59.289279+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9612,"log_dups_size":0,"ondisk_log_size":9612,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20256370,"num_objects":2473,"num_object_clones":0,"num_object_copies":9892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9610,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":417}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9488","reported_seq":9579,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.584996+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:13:59.584996+0000","last_peered":"2024-09-15T03:13:59.584996+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:13:59.584996+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9488,"log_dups_size":0,"ondisk_log_size":9488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20327652,"num_objects":2482,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9509","reported_seq":9628,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.543985+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:13:59.543985+0000","last_peered":"2024-09-15T03:13:59.543985+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:13:59.543985+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9509,"log_dups_size":0,"ondisk_log_size":9509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141682,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9507,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":417}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9508","reported_seq":9602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.552735+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:13:59.552735+0000","last_peered":"2024-09-15T03:13:59.552735+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:13:59.552735+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9508,"log_dups_size":0,"ondisk_log_size":9508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19452331,"num_objects":2375,"num_object_clones":0,"num_object_copies":9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":313}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9413","reported_seq":13454,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:13:56.088799+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:13:56.088799+0000","last_peered":"2024-09-15T03:13:56.088799+0000","last_clean":"2024-09-15T03:13:56.088799+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:13:56.088799+0000","last_undegraded":"2024-09-15T03:13:56.088799+0000","last_fullsized":"2024-09-15T03:13:56.088799+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9413,"log_dups_size":0,"ondisk_log_size":9413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20364089,"num_objects":2486,"num_object_clones":0,"num_object_copies":9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9412,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9584","reported_seq":12276,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.740014+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:13:59.740014+0000","last_peered":"2024-09-15T03:13:59.740014+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:13:59.740014+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20880185,"num_objects":2549,"num_object_clones":0,"num_object_copies":10196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1306,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1814}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9295","reported_seq":9395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.643986+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:13:59.643986+0000","last_peered":"2024-09-15T03:13:59.643986+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:13:59.643986+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9295,"log_dups_size":0,"ondisk_log_size":9295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19551858,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9293,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":374}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9454","reported_seq":9557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.760213+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:13:59.760213+0000","last_peered":"2024-09-15T03:13:59.760213+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:13:59.760213+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19895922,"num_objects":2429,"num_object_clones":0,"num_object_copies":9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9452,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":452}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9558","reported_seq":9670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:13:59.440846+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:13:59.440846+0000","last_peered":"2024-09-15T03:13:59.440846+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:13:59.440846+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20076146,"num_objects":2451,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9556,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":322329172,"num_objects":39352,"num_object_clones":0,"num_object_copies":157408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151355,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5501,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":645754880,"data_stored":645754880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151382,"ondisk_log_size":151382,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":463564,"kb_used_data":157380,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93908276,"statfs":{"total":96636764160,"available":96162074624,"internally_reserved":0,"allocated":161157120,"data_stored":160981654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":6.218}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.248},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.38800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":1.905},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.42099999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471460,"kb_used_data":157468,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":93900380,"statfs":{"total":96636764160,"available":96153989120,"internally_reserved":0,"allocated":161247232,"data_stored":161074912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,1,2,3,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":12.497999999999999},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":12.573}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.34699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":5.0119999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466148,"kb_used_data":159948,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93905692,"statfs":{"total":96636764160,"available":96159428608,"internally_reserved":0,"allocated":163786752,"data_stored":163611286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":28.076000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":0.32800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":28.145}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":1.5580000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455360,"kb_used_data":157560,"kb_used_omap":4,"kb_used_meta":297787,"kb_avail":93916480,"statfs":{"total":96636764160,"available":96170475520,"internally_reserved":0,"allocated":161341440,"data_stored":161165974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":304934465},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,2,0,4,3,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":7.9290000000000003},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":3.9820000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":9.0860000000000003},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":4.0179999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160931840,"data_stored":160931840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":163438592,"data_stored":163438592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":160702464,"data_stored":160702464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":160681984,"data_stored":160681984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:05.080 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:07.225 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:07.263 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:07.289 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-09-15T03:14:06.888403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":313530964,"num_objects":38278,"num_object_clones":0,"num_object_copies":153112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152429,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5555,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152456,"ondisk_log_size":152456,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694672,"kb_used_data":613680,"kb_used_omap":14,"kb_used_meta":2080945,"kb_avail":374792688,"statfs":{"total":386547056640,"available":383787712512,"internally_reserved":0,"allocated":628408320,"data_stored":627709602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130887707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,13,13,13,8,8,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16539648,"num_objects":-2019,"num_object_clones":0,"num_object_copies":-8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008525"},"pg_stats":[{"pgid":"1.f","version":"35'9468","reported_seq":9534,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.667424+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:04.667424+0000","last_peered":"2024-09-15T03:14:04.667424+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:04.667424+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20004864,"num_objects":2442,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":476}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9139","reported_seq":9233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.448723+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:04.448723+0000","last_peered":"2024-09-15T03:14:04.448723+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:04.448723+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18732658,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9745","reported_seq":9853,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.781064+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:04.781064+0000","last_peered":"2024-09-15T03:14:04.781064+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:04.781064+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9745,"log_dups_size":0,"ondisk_log_size":9745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20274001,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9744,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9228","reported_seq":13200,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:06.305958+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:06.305958+0000","last_peered":"2024-09-15T03:14:06.305958+0000","last_clean":"2024-09-15T03:14:06.305958+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:06.305958+0000","last_undegraded":"2024-09-15T03:14:06.305958+0000","last_fullsized":"2024-09-15T03:14:06.305958+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9228,"log_dups_size":0,"ondisk_log_size":9228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19658354,"num_objects":2400,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9649","reported_seq":9755,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.688873+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:04.688873+0000","last_peered":"2024-09-15T03:14:04.688873+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:04.688873+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19920498,"num_objects":2432,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9666","reported_seq":9750,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.626013+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:04.626013+0000","last_peered":"2024-09-15T03:14:04.626013+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:04.626013+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9575","reported_seq":9694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.465610+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:04.465610+0000","last_peered":"2024-09-15T03:14:04.465610+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:04.465610+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9575,"log_dups_size":0,"ondisk_log_size":9575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19519090,"num_objects":2383,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":357}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9663","reported_seq":9780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.289422+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:04.289422+0000","last_peered":"2024-09-15T03:14:04.289422+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:04.289422+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9663,"log_dups_size":0,"ondisk_log_size":9663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19838578,"num_objects":2422,"num_object_clones":0,"num_object_copies":9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9536","reported_seq":9627,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.509879+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:04.509879+0000","last_peered":"2024-09-15T03:14:04.509879+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:04.509879+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9536,"log_dups_size":0,"ondisk_log_size":9536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19934436,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9568","reported_seq":9686,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.753217+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:04.753217+0000","last_peered":"2024-09-15T03:14:04.753217+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:04.753217+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9568,"log_dups_size":0,"ondisk_log_size":9568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19658354,"num_objects":2400,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9566,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9558","reported_seq":9652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.588634+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:04.588634+0000","last_peered":"2024-09-15T03:14:04.588634+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:04.588634+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042731,"num_objects":2325,"num_object_clones":0,"num_object_copies":9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9555,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9534","reported_seq":13575,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:06.622702+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:06.622702+0000","last_peered":"2024-09-15T03:14:06.622702+0000","last_clean":"2024-09-15T03:14:06.622702+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:06.622702+0000","last_undegraded":"2024-09-15T03:14:06.622702+0000","last_fullsized":"2024-09-15T03:14:06.622702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19372857,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9645","reported_seq":12442,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.753195+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:04.753195+0000","last_peered":"2024-09-15T03:14:04.753195+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:04.753195+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20380473,"num_objects":2488,"num_object_clones":0,"num_object_copies":9952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9644,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9353","reported_seq":9453,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.739800+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:04.739800+0000","last_peered":"2024-09-15T03:14:04.739800+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:04.739800+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9353,"log_dups_size":0,"ondisk_log_size":9353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19076722,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":316}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9525","reported_seq":9632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.700779+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:04.700779+0000","last_peered":"2024-09-15T03:14:04.700779+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:04.700779+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9525,"log_dups_size":0,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19314290,"num_objects":2358,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":381}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9604","reported_seq":9717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.444859+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:04.444859+0000","last_peered":"2024-09-15T03:14:04.444859+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:04.444859+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9604,"log_dups_size":0,"ondisk_log_size":9604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19699314,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9602,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":313530964,"num_objects":38278,"num_object_clones":0,"num_object_copies":153112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152429,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5555,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626237440,"data_stored":626237440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152456,"ondisk_log_size":152456,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752344,"kb_used_data":153616,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93619496,"statfs":{"total":96636764160,"available":95866363904,"internally_reserved":0,"allocated":157302784,"data_stored":157127318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,3,1,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.28699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.504}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.53200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743672,"kb_used_data":153776,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93628168,"statfs":{"total":96636764160,"available":95875244032,"internally_reserved":0,"allocated":157466624,"data_stored":157294304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,5,5,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.308},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.29699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.39100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454472,"kb_used_data":152448,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93917368,"statfs":{"total":96636764160,"available":96171384832,"internally_reserved":0,"allocated":156106752,"data_stored":155931286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,3,2,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":2.4159999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":11.853},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":8.9000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.26000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744184,"kb_used_data":153840,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93627656,"statfs":{"total":96636764160,"available":95874719744,"internally_reserved":0,"allocated":157532160,"data_stored":157356694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,7,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.41999999999999998},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.307},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.40699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157024256,"data_stored":157024256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155643904,"data_stored":155643904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":156753920,"data_stored":156753920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":156815360,"data_stored":156815360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:07.291 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:08.790 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:08.835 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:08.852 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-09-15T03:14:06.888403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":313530964,"num_objects":38278,"num_object_clones":0,"num_object_copies":153112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152429,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5555,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152456,"ondisk_log_size":152456,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694672,"kb_used_data":613680,"kb_used_omap":14,"kb_used_meta":2080945,"kb_avail":374792688,"statfs":{"total":386547056640,"available":383787712512,"internally_reserved":0,"allocated":628408320,"data_stored":627709602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130887707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,13,13,13,8,8,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16539648,"num_objects":-2019,"num_object_clones":0,"num_object_copies":-8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2019,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008525"},"pg_stats":[{"pgid":"1.f","version":"35'9468","reported_seq":9534,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.667424+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:04.667424+0000","last_peered":"2024-09-15T03:14:04.667424+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:04.667424+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9468,"log_dups_size":0,"ondisk_log_size":9468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20004864,"num_objects":2442,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":476}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9139","reported_seq":9233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.448723+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:04.448723+0000","last_peered":"2024-09-15T03:14:04.448723+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:04.448723+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18732658,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9137,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":303}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9745","reported_seq":9853,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.781064+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:04.781064+0000","last_peered":"2024-09-15T03:14:04.781064+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:04.781064+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9745,"log_dups_size":0,"ondisk_log_size":9745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20274001,"num_objects":2476,"num_object_clones":0,"num_object_copies":9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9744,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9228","reported_seq":13200,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:06.305958+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:06.305958+0000","last_peered":"2024-09-15T03:14:06.305958+0000","last_clean":"2024-09-15T03:14:06.305958+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:06.305958+0000","last_undegraded":"2024-09-15T03:14:06.305958+0000","last_fullsized":"2024-09-15T03:14:06.305958+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9228,"log_dups_size":0,"ondisk_log_size":9228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19658354,"num_objects":2400,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9649","reported_seq":9755,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.688873+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:04.688873+0000","last_peered":"2024-09-15T03:14:04.688873+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:04.688873+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19920498,"num_objects":2432,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":348}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9666","reported_seq":9750,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.626013+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:04.626013+0000","last_peered":"2024-09-15T03:14:04.626013+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:04.626013+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19103744,"num_objects":2332,"num_object_clones":0,"num_object_copies":9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9666,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9575","reported_seq":9694,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.465610+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:04.465610+0000","last_peered":"2024-09-15T03:14:04.465610+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:04.465610+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9575,"log_dups_size":0,"ondisk_log_size":9575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19519090,"num_objects":2383,"num_object_clones":0,"num_object_copies":9532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":357}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9663","reported_seq":9780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.289422+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:04.289422+0000","last_peered":"2024-09-15T03:14:04.289422+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:04.289422+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9663,"log_dups_size":0,"ondisk_log_size":9663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19838578,"num_objects":2422,"num_object_clones":0,"num_object_copies":9688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9661,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9536","reported_seq":9627,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.509879+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:04.509879+0000","last_peered":"2024-09-15T03:14:04.509879+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:04.509879+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9536,"log_dups_size":0,"ondisk_log_size":9536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19934436,"num_objects":2434,"num_object_clones":0,"num_object_copies":9736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9568","reported_seq":9686,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.753217+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:04.753217+0000","last_peered":"2024-09-15T03:14:04.753217+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:04.753217+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9568,"log_dups_size":0,"ondisk_log_size":9568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19658354,"num_objects":2400,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9566,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":358}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9558","reported_seq":9652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.588634+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:04.588634+0000","last_peered":"2024-09-15T03:14:04.588634+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:04.588634+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042731,"num_objects":2325,"num_object_clones":0,"num_object_copies":9300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9555,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":263}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9534","reported_seq":13575,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:06.622702+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:06.622702+0000","last_peered":"2024-09-15T03:14:06.622702+0000","last_clean":"2024-09-15T03:14:06.622702+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:06.622702+0000","last_undegraded":"2024-09-15T03:14:06.622702+0000","last_fullsized":"2024-09-15T03:14:06.622702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19372857,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9645","reported_seq":12442,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.753195+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:04.753195+0000","last_peered":"2024-09-15T03:14:04.753195+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:04.753195+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9645,"log_dups_size":0,"ondisk_log_size":9645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20380473,"num_objects":2488,"num_object_clones":0,"num_object_copies":9952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9644,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9353","reported_seq":9453,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.739800+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:04.739800+0000","last_peered":"2024-09-15T03:14:04.739800+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:04.739800+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9353,"log_dups_size":0,"ondisk_log_size":9353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19076722,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":316}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9525","reported_seq":9632,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.700779+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:04.700779+0000","last_peered":"2024-09-15T03:14:04.700779+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:04.700779+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9525,"log_dups_size":0,"ondisk_log_size":9525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19314290,"num_objects":2358,"num_object_clones":0,"num_object_copies":9432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":381}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9604","reported_seq":9717,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:04.444859+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:04.444859+0000","last_peered":"2024-09-15T03:14:04.444859+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:04.444859+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9604,"log_dups_size":0,"ondisk_log_size":9604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19699314,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9602,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":313530964,"num_objects":38278,"num_object_clones":0,"num_object_copies":153112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152429,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5555,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626237440,"data_stored":626237440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152456,"ondisk_log_size":152456,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752344,"kb_used_data":153616,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93619496,"statfs":{"total":96636764160,"available":95866363904,"internally_reserved":0,"allocated":157302784,"data_stored":157127318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,3,1,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.28699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.504}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.53200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743672,"kb_used_data":153776,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93628168,"statfs":{"total":96636764160,"available":95875244032,"internally_reserved":0,"allocated":157466624,"data_stored":157294304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,3,5,5,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.308},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.29699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.39100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454472,"kb_used_data":152448,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93917368,"statfs":{"total":96636764160,"available":96171384832,"internally_reserved":0,"allocated":156106752,"data_stored":155931286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,3,2,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":2.4159999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":11.853},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":8.9000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.26000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744184,"kb_used_data":153840,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93627656,"statfs":{"total":96636764160,"available":95874719744,"internally_reserved":0,"allocated":157532160,"data_stored":157356694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,7,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.44700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.41999999999999998},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.307},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.40699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":157024256,"data_stored":157024256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155643904,"data_stored":155643904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":156753920,"data_stored":156753920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":156815360,"data_stored":156815360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:08.854 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:14:08.854 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9468", 'reported_seq': 9534, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.667424+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:14:04.667424+0000', 'last_peered': '2024-09-15T03:14:04.667424+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:14:04.667424+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9468, 'log_dups_size': 0, 'ondisk_log_size': 9468, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20004864, 'num_objects': 2442, 'num_object_clones': 0, 'num_object_copies': 9768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9468, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 476}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:08.854 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:14:08.854 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9139", 'reported_seq': 9233, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.448723+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:14:04.448723+0000', 'last_peered': '2024-09-15T03:14:04.448723+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:14:04.448723+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9139, 'log_dups_size': 0, 'ondisk_log_size': 9139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18732658, 'num_objects': 2287, 'num_object_clones': 0, 'num_object_copies': 9148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9137, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:08.854 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9745", 'reported_seq': 9853, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.781064+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:14:04.781064+0000', 'last_peered': '2024-09-15T03:14:04.781064+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:14:04.781064+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9745, 'log_dups_size': 0, 'ondisk_log_size': 9745, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20274001, 'num_objects': 2476, 'num_object_clones': 0, 'num_object_copies': 9904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2476, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9744, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 385}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9649", 'reported_seq': 9755, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.688873+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:14:04.688873+0000', 'last_peered': '2024-09-15T03:14:04.688873+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:14:04.688873+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9649, 'log_dups_size': 0, 'ondisk_log_size': 9649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19920498, 'num_objects': 2432, 'num_object_clones': 0, 'num_object_copies': 9728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9647, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 348}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9666", 'reported_seq': 9750, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.626013+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:14:04.626013+0000', 'last_peered': '2024-09-15T03:14:04.626013+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:14:04.626013+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9666, 'log_dups_size': 0, 'ondisk_log_size': 9666, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19103744, 'num_objects': 2332, 'num_object_clones': 0, 'num_object_copies': 9328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9666, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9575", 'reported_seq': 9694, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.465610+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:14:04.465610+0000', 'last_peered': '2024-09-15T03:14:04.465610+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:14:04.465610+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9575, 'log_dups_size': 0, 'ondisk_log_size': 9575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19519090, 'num_objects': 2383, 'num_object_clones': 0, 'num_object_copies': 9532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9573, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 357}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9663", 'reported_seq': 9780, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.289422+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:14:04.289422+0000', 'last_peered': '2024-09-15T03:14:04.289422+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:14:04.289422+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9663, 'log_dups_size': 0, 'ondisk_log_size': 9663, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19838578, 'num_objects': 2422, 'num_object_clones': 0, 'num_object_copies': 9688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2422, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9661, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:14:08.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9536", 'reported_seq': 9627, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.509879+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:14:04.509879+0000', 'last_peered': '2024-09-15T03:14:04.509879+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:14:04.509879+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9536, 'log_dups_size': 0, 'ondisk_log_size': 9536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19934436, 'num_objects': 2434, 'num_object_clones': 0, 'num_object_copies': 9736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9533, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 407}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9568", 'reported_seq': 9686, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.753217+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:14:04.753217+0000', 'last_peered': '2024-09-15T03:14:04.753217+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:14:04.753217+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9568, 'log_dups_size': 0, 'ondisk_log_size': 9568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19658354, 'num_objects': 2400, 'num_object_clones': 0, 'num_object_copies': 9600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9566, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 358}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9558", 'reported_seq': 9652, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.588634+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:14:04.588634+0000', 'last_peered': '2024-09-15T03:14:04.588634+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:14:04.588634+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9558, 'log_dups_size': 0, 'ondisk_log_size': 9558, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19042731, 'num_objects': 2325, 'num_object_clones': 0, 'num_object_copies': 9300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2325, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9555, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 263}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9645", 'reported_seq': 12442, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.753195+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:14:04.753195+0000', 'last_peered': '2024-09-15T03:14:04.753195+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:14:04.753195+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9645, 'log_dups_size': 0, 'ondisk_log_size': 9645, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20380473, 'num_objects': 2488, 'num_object_clones': 0, 'num_object_copies': 9952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 681, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9644, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1360, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1807}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9353", 'reported_seq': 9453, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.739800+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:14:04.739800+0000', 'last_peered': '2024-09-15T03:14:04.739800+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:14:04.739800+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9353, 'log_dups_size': 0, 'ondisk_log_size': 9353, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19076722, 'num_objects': 2329, 'num_object_clones': 0, 'num_object_copies': 9316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9351, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 316}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:08.856 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:14:08.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9525", 'reported_seq': 9632, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.700779+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:14:04.700779+0000', 'last_peered': '2024-09-15T03:14:04.700779+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:14:04.700779+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9525, 'log_dups_size': 0, 'ondisk_log_size': 9525, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19314290, 'num_objects': 2358, 'num_object_clones': 0, 'num_object_copies': 9432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9523, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:08.857 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:14:08.857 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9604", 'reported_seq': 9717, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:04.444859+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:14:04.444859+0000', 'last_peered': '2024-09-15T03:14:04.444859+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:14:04.444859+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9604, 'log_dups_size': 0, 'ondisk_log_size': 9604, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19699314, 'num_objects': 2405, 'num_object_clones': 0, 'num_object_copies': 9620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9602, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:08.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:14:10.834 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:10.834 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27129/153112 objects degraded (17.718%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":573,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":38278,"data_bytes":313530964,"bytes_used":2759344128,"bytes_avail":383787712512,"bytes_total":386547056640,"degraded_objects":27129,"degraded_total":153112,"degraded_ratio":0.17718402215371754,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":158},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:14:10.904 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:14:10.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:12.724 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:12.729 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:12.748 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-15T03:14:10.889313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306764372,"num_objects":37452,"num_object_clones":0,"num_object_copies":149808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37452,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153255,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5611,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153282,"ondisk_log_size":153282,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2687804,"kb_used_data":602640,"kb_used_omap":14,"kb_used_meta":2085105,"kb_avail":374799556,"statfs":{"total":386547056640,"available":383794745344,"internally_reserved":0,"allocated":617103360,"data_stored":616404642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135147547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,7,5,12,9,11,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22323200,"num_objects":-2725,"num_object_clones":0,"num_object_copies":-10900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008432"},"pg_stats":[{"pgid":"1.f","version":"35'9533","reported_seq":9599,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.779506+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:09.779506+0000","last_peered":"2024-09-15T03:14:09.779506+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:09.779506+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9533,"log_dups_size":0,"ondisk_log_size":9533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19472384,"num_objects":2377,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9191","reported_seq":9285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.514874+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:09.514874+0000","last_peered":"2024-09-15T03:14:09.514874+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:09.514874+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9191,"log_dups_size":0,"ondisk_log_size":9191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18306674,"num_objects":2235,"num_object_clones":0,"num_object_copies":8940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9189,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9802","reported_seq":9910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.652951+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:09.652951+0000","last_peered":"2024-09-15T03:14:09.652951+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:09.652951+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9802,"log_dups_size":0,"ondisk_log_size":9802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19807057,"num_objects":2419,"num_object_clones":0,"num_object_copies":9676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2419,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9801,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":328}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9228","reported_seq":13200,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:06.305958+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:06.305958+0000","last_peered":"2024-09-15T03:14:06.305958+0000","last_clean":"2024-09-15T03:14:06.305958+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:06.305958+0000","last_undegraded":"2024-09-15T03:14:06.305958+0000","last_fullsized":"2024-09-15T03:14:06.305958+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9228,"log_dups_size":0,"ondisk_log_size":9228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19658354,"num_objects":2400,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9226,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9708","reported_seq":9814,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.832255+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:09.832255+0000","last_peered":"2024-09-15T03:14:09.832255+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:09.832255+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9708,"log_dups_size":0,"ondisk_log_size":9708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19437170,"num_objects":2373,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9729","reported_seq":9813,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.825413+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:09.825413+0000","last_peered":"2024-09-15T03:14:09.825413+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:09.825413+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9729,"log_dups_size":0,"ondisk_log_size":9729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18587648,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9729,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9631","reported_seq":9752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.454660+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:09.454660+0000","last_peered":"2024-09-15T03:14:09.454660+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:09.454660+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9631,"log_dups_size":0,"ondisk_log_size":9631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19060338,"num_objects":2327,"num_object_clones":0,"num_object_copies":9308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9629,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":301}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9728","reported_seq":9845,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.417960+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:09.417960+0000","last_peered":"2024-09-15T03:14:09.417960+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:09.417960+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9728,"log_dups_size":0,"ondisk_log_size":9728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19306098,"num_objects":2357,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9726,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9598","reported_seq":9689,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.891497+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:09.891497+0000","last_peered":"2024-09-15T03:14:09.891497+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:09.891497+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9598,"log_dups_size":0,"ondisk_log_size":9598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19426532,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9595,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9626","reported_seq":9745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.679004+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:09.679004+0000","last_peered":"2024-09-15T03:14:09.679004+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:09.679004+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19183218,"num_objects":2342,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":300}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9628","reported_seq":9721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.903523+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:09.903523+0000","last_peered":"2024-09-15T03:14:09.903523+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:09.903523+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9628,"log_dups_size":0,"ondisk_log_size":9628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18469291,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9625,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9534","reported_seq":13575,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:06.622702+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:06.622702+0000","last_peered":"2024-09-15T03:14:06.622702+0000","last_clean":"2024-09-15T03:14:06.622702+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:06.622702+0000","last_undegraded":"2024-09-15T03:14:06.622702+0000","last_fullsized":"2024-09-15T03:14:06.622702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19372857,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9701","reported_seq":12612,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.931375+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:09.931375+0000","last_peered":"2024-09-15T03:14:09.931375+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:09.931375+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9701,"log_dups_size":0,"ondisk_log_size":9701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19921721,"num_objects":2432,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9700,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9409","reported_seq":9509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.608078+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:09.608078+0000","last_peered":"2024-09-15T03:14:09.608078+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:09.608078+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9409,"log_dups_size":0,"ondisk_log_size":9409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18617970,"num_objects":2273,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9407,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":260}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9574","reported_seq":9681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.511369+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:09.511369+0000","last_peered":"2024-09-15T03:14:09.511369+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:09.511369+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9574,"log_dups_size":0,"ondisk_log_size":9574,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18912882,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9572,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":332}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9662","reported_seq":9775,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:09.592698+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:09.592698+0000","last_peered":"2024-09-15T03:14:09.592698+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:09.592698+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9662,"log_dups_size":0,"ondisk_log_size":9662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19224178,"num_objects":2347,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9660,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":288}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":306764372,"num_objects":37452,"num_object_clones":0,"num_object_copies":149808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37452,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153255,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5611,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":614739968,"data_stored":614739968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153282,"ondisk_log_size":153282,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740452,"kb_used_data":150044,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93631388,"statfs":{"total":96636764160,"available":95878541312,"internally_reserved":0,"allocated":153645056,"data_stored":153469590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,3,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":4.1790000000000003},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.23699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":3.98},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":4.4950000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":6.0819999999999999},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":4.2199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744132,"kb_used_data":150076,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93627708,"statfs":{"total":96636764160,"available":95874772992,"internally_reserved":0,"allocated":153677824,"data_stored":153505504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.252}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.4049999999999998},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":3.4329999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":3.4609999999999999},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":454472,"kb_used_data":152448,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93917368,"statfs":{"total":96636764160,"available":96171384832,"internally_reserved":0,"allocated":156106752,"data_stored":155931286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,0,3,2,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3160000000000001,"15min":1.8460000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":4.476,"5min":11.047000000000001,"15min":18.091000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.96199999999999997,"5min":2.0179999999999998,"15min":1.6639999999999999},"min":{"1min":0.216,"5min":0.191,"15min":0.187},"max":{"1min":6.5430000000000001,"5min":29.294,"15min":29.294},"last":2.4159999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":2.0270000000000001,"15min":1.5740000000000001},"min":{"1min":0.27600000000000002,"5min":0.219,"15min":0.219},"max":{"1min":8.1059999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":11.853},{"interface":"front","average":{"1min":1.829,"5min":2.6440000000000001,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":5.9320000000000004,"5min":20.055,"15min":20.055},"last":8.9000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.8759999999999999,"15min":1.6910000000000001},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":4.2930000000000001,"5min":11.413,"15min":13.414999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.0190000000000001,"15min":1.5},"min":{"1min":0.22800000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":11.052,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.26000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748748,"kb_used_data":150072,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93623092,"statfs":{"total":96636764160,"available":95870046208,"internally_reserved":0,"allocated":153673728,"data_stored":153498262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,4,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":3.7250000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":1.151},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.88}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.33100000000000002},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.60699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":153174016,"data_stored":153174016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155643904,"data_stored":155643904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":152940544,"data_stored":152940544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":152981504,"data_stored":152981504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:15.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:17.131 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:17.134 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:17.146 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-15T03:14:16.890833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298220116,"num_objects":36409,"num_object_clones":0,"num_object_copies":145636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154298,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5665,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154325,"ondisk_log_size":154325,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690396,"kb_used_data":584444,"kb_used_omap":14,"kb_used_meta":2105905,"kb_avail":374796964,"statfs":{"total":386547056640,"available":383792091136,"internally_reserved":0,"allocated":598470656,"data_stored":597771938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2156446747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,8,14,15,15,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18448384,"num_objects":-2252,"num_object_clones":0,"num_object_copies":-9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2252,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005952"},"pg_stats":[{"pgid":"1.f","version":"35'9594","reported_seq":9660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.687407+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:14.687407+0000","last_peered":"2024-09-15T03:14:14.687407+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:14.687407+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9594,"log_dups_size":0,"ondisk_log_size":9594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18972672,"num_objects":2316,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9594,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9242","reported_seq":9336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.487759+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:14.487759+0000","last_peered":"2024-09-15T03:14:14.487759+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:14.487759+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9242,"log_dups_size":0,"ondisk_log_size":9242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17888882,"num_objects":2184,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9240,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9871","reported_seq":9979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.932261+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:14.932261+0000","last_peered":"2024-09-15T03:14:14.932261+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:14.932261+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19241809,"num_objects":2350,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9870,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9348","reported_seq":13320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:16.681567+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:16.681567+0000","last_peered":"2024-09-15T03:14:16.681567+0000","last_clean":"2024-09-15T03:14:16.681567+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:16.681567+0000","last_undegraded":"2024-09-15T03:14:16.681567+0000","last_fullsized":"2024-09-15T03:14:16.681567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18675314,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9759","reported_seq":9865,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.848379+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:14.848379+0000","last_peered":"2024-09-15T03:14:14.848379+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:14.848379+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19019378,"num_objects":2322,"num_object_clones":0,"num_object_copies":9288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9757,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9792","reported_seq":9876,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.747740+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:14.747740+0000","last_peered":"2024-09-15T03:14:14.747740+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:14.747740+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18071552,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9697","reported_seq":9818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.702972+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:14.702972+0000","last_peered":"2024-09-15T03:14:14.702972+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:14.702972+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9697,"log_dups_size":0,"ondisk_log_size":9697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18519666,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":235}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9791","reported_seq":9908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.715613+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:14.715613+0000","last_peered":"2024-09-15T03:14:14.715613+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:14.715613+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9791,"log_dups_size":0,"ondisk_log_size":9791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18790002,"num_objects":2294,"num_object_clones":0,"num_object_copies":9176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9649","reported_seq":9740,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.791289+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:14.791289+0000","last_peered":"2024-09-15T03:14:14.791289+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:14.791289+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19008740,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9646,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9685","reported_seq":9804,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.811068+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:14.811068+0000","last_peered":"2024-09-15T03:14:14.811068+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:14.811068+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18699890,"num_objects":2283,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9684","reported_seq":9778,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.726284+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:14.726284+0000","last_peered":"2024-09-15T03:14:14.726284+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:14.726284+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18010539,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9681,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":137}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9635","reported_seq":13676,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:16.527577+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:16.527577+0000","last_peered":"2024-09-15T03:14:16.527577+0000","last_clean":"2024-09-15T03:14:16.527577+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:16.527577+0000","last_undegraded":"2024-09-15T03:14:16.527577+0000","last_fullsized":"2024-09-15T03:14:16.527577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9635,"log_dups_size":0,"ondisk_log_size":9635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18545465,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9755","reported_seq":12775,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.950885+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:14.950885+0000","last_peered":"2024-09-15T03:14:14.950885+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:14.950885+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19479353,"num_objects":2378,"num_object_clones":0,"num_object_copies":9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9754,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1470,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9472","reported_seq":9572,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.870440+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:14.870440+0000","last_peered":"2024-09-15T03:14:14.870440+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:14.870440+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9472,"log_dups_size":0,"ondisk_log_size":9472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18101874,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9470,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9636","reported_seq":9741,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.783013+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:14.783013+0000","last_peered":"2024-09-15T03:14:14.783013+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:14.783013+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18404978,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":270}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9715","reported_seq":9828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.536659+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:14.536659+0000","last_peered":"2024-09-15T03:14:14.536659+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:14.536659+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9715,"log_dups_size":0,"ondisk_log_size":9715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18790002,"num_objects":2294,"num_object_clones":0,"num_object_copies":9176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":298220116,"num_objects":36409,"num_object_clones":0,"num_object_copies":145636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154298,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5665,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595701760,"data_stored":595701760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154325,"ondisk_log_size":154325,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740992,"kb_used_data":146424,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93630848,"statfs":{"total":96636764160,"available":95877988352,"internally_reserved":0,"allocated":149938176,"data_stored":149762710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,6,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.33000000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.93100000000000005},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.96799999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744556,"kb_used_data":146340,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93627284,"statfs":{"total":96636764160,"available":95874338816,"internally_reserved":0,"allocated":149852160,"data_stored":149679840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,6,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":2.8860000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.316},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.36799999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":2.8519999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455592,"kb_used_data":145248,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93916248,"statfs":{"total":96636764160,"available":96170237952,"internally_reserved":0,"allocated":148733952,"data_stored":148558486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.224}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.90300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.44900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749256,"kb_used_data":146432,"kb_used_omap":4,"kb_used_meta":602811,"kb_avail":93622584,"statfs":{"total":96636764160,"available":95869526016,"internally_reserved":0,"allocated":149946368,"data_stored":149770902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617279041},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,0,2,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.373}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":12.042}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.47499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149274624,"data_stored":149274624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148090880,"data_stored":148090880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149164032,"data_stored":149164032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149172224,"data_stored":149172224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:17.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:17.899 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:17.921 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:17.964 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-15T03:14:16.890833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298220116,"num_objects":36409,"num_object_clones":0,"num_object_copies":145636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154298,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5665,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154325,"ondisk_log_size":154325,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690396,"kb_used_data":584444,"kb_used_omap":14,"kb_used_meta":2105905,"kb_avail":374796964,"statfs":{"total":386547056640,"available":383792091136,"internally_reserved":0,"allocated":598470656,"data_stored":597771938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2156446747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,8,14,15,15,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18448384,"num_objects":-2252,"num_object_clones":0,"num_object_copies":-9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2252,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005952"},"pg_stats":[{"pgid":"1.f","version":"35'9594","reported_seq":9660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.687407+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:14.687407+0000","last_peered":"2024-09-15T03:14:14.687407+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:14.687407+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9594,"log_dups_size":0,"ondisk_log_size":9594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18972672,"num_objects":2316,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9594,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9242","reported_seq":9336,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.487759+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:14.487759+0000","last_peered":"2024-09-15T03:14:14.487759+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:14.487759+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9242,"log_dups_size":0,"ondisk_log_size":9242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17888882,"num_objects":2184,"num_object_clones":0,"num_object_copies":8736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9240,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9871","reported_seq":9979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.932261+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:14.932261+0000","last_peered":"2024-09-15T03:14:14.932261+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:14.932261+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19241809,"num_objects":2350,"num_object_clones":0,"num_object_copies":9400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9870,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9348","reported_seq":13320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:16.681567+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:16.681567+0000","last_peered":"2024-09-15T03:14:16.681567+0000","last_clean":"2024-09-15T03:14:16.681567+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:16.681567+0000","last_undegraded":"2024-09-15T03:14:16.681567+0000","last_fullsized":"2024-09-15T03:14:16.681567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18675314,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9759","reported_seq":9865,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.848379+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:14.848379+0000","last_peered":"2024-09-15T03:14:14.848379+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:14.848379+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9759,"log_dups_size":0,"ondisk_log_size":9759,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19019378,"num_objects":2322,"num_object_clones":0,"num_object_copies":9288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9757,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9792","reported_seq":9876,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.747740+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:14.747740+0000","last_peered":"2024-09-15T03:14:14.747740+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:14.747740+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18071552,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":168}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9697","reported_seq":9818,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.702972+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:14.702972+0000","last_peered":"2024-09-15T03:14:14.702972+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:14.702972+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9697,"log_dups_size":0,"ondisk_log_size":9697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18519666,"num_objects":2261,"num_object_clones":0,"num_object_copies":9044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9695,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":235}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9791","reported_seq":9908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.715613+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:14.715613+0000","last_peered":"2024-09-15T03:14:14.715613+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:14.715613+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9791,"log_dups_size":0,"ondisk_log_size":9791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18790002,"num_objects":2294,"num_object_clones":0,"num_object_copies":9176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9789,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9649","reported_seq":9740,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.791289+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:14.791289+0000","last_peered":"2024-09-15T03:14:14.791289+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:14.791289+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19008740,"num_objects":2321,"num_object_clones":0,"num_object_copies":9284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9646,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9685","reported_seq":9804,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.811068+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:14.811068+0000","last_peered":"2024-09-15T03:14:14.811068+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:14.811068+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18699890,"num_objects":2283,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9684","reported_seq":9778,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.726284+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:14.726284+0000","last_peered":"2024-09-15T03:14:14.726284+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:14.726284+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9684,"log_dups_size":0,"ondisk_log_size":9684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18010539,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9681,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":137}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9635","reported_seq":13676,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:16.527577+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:16.527577+0000","last_peered":"2024-09-15T03:14:16.527577+0000","last_clean":"2024-09-15T03:14:16.527577+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:16.527577+0000","last_undegraded":"2024-09-15T03:14:16.527577+0000","last_fullsized":"2024-09-15T03:14:16.527577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9635,"log_dups_size":0,"ondisk_log_size":9635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18545465,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9755","reported_seq":12775,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.950885+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:14.950885+0000","last_peered":"2024-09-15T03:14:14.950885+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:14.950885+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19479353,"num_objects":2378,"num_object_clones":0,"num_object_copies":9512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9754,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1470,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1807}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9472","reported_seq":9572,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.870440+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:14.870440+0000","last_peered":"2024-09-15T03:14:14.870440+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:14.870440+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9472,"log_dups_size":0,"ondisk_log_size":9472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18101874,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9470,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":197}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9636","reported_seq":9741,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.783013+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:14.783013+0000","last_peered":"2024-09-15T03:14:14.783013+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:14.783013+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9636,"log_dups_size":0,"ondisk_log_size":9636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18404978,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":270}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9715","reported_seq":9828,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:14.536659+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:14.536659+0000","last_peered":"2024-09-15T03:14:14.536659+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:14.536659+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9715,"log_dups_size":0,"ondisk_log_size":9715,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18790002,"num_objects":2294,"num_object_clones":0,"num_object_copies":9176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":298220116,"num_objects":36409,"num_object_clones":0,"num_object_copies":145636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27019,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154298,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5665,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595701760,"data_stored":595701760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154325,"ondisk_log_size":154325,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740992,"kb_used_data":146424,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93630848,"statfs":{"total":96636764160,"available":95877988352,"internally_reserved":0,"allocated":149938176,"data_stored":149762710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,6,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":0.33000000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.93100000000000005},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.96799999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744556,"kb_used_data":146340,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93627284,"statfs":{"total":96636764160,"available":95874338816,"internally_reserved":0,"allocated":149852160,"data_stored":149679840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,6,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":2.8860000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7829999999999999,"5min":1.8140000000000001,"15min":1.357},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":32.295999999999999,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.316},{"interface":"front","average":{"1min":2.827,"5min":1.7250000000000001,"15min":1.4119999999999999},"min":{"1min":0.28299999999999997,"5min":0.248,"15min":0.20999999999999999},"max":{"1min":32.340000000000003,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.36799999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":2.8519999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455592,"kb_used_data":145248,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93916248,"statfs":{"total":96636764160,"available":96170237952,"internally_reserved":0,"allocated":148733952,"data_stored":148558486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.224}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.90300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.44900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":749256,"kb_used_data":146432,"kb_used_omap":4,"kb_used_meta":602811,"kb_avail":93622584,"statfs":{"total":96636764160,"available":95869526016,"internally_reserved":0,"allocated":149946368,"data_stored":149770902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617279041},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,0,2,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.373}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":12.042}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.47499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149274624,"data_stored":149274624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148090880,"data_stored":148090880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149164032,"data_stored":149164032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":149172224,"data_stored":149172224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:17.965 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:14:17.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9594", 'reported_seq': 9660, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.687407+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:14:14.687407+0000', 'last_peered': '2024-09-15T03:14:14.687407+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:14:14.687407+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9594, 'log_dups_size': 0, 'ondisk_log_size': 9594, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18972672, 'num_objects': 2316, 'num_object_clones': 0, 'num_object_copies': 9264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9594, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 350}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:17.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:14:17.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9242", 'reported_seq': 9336, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.487759+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:14:14.487759+0000', 'last_peered': '2024-09-15T03:14:14.487759+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:14:14.487759+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9242, 'log_dups_size': 0, 'ondisk_log_size': 9242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17888882, 'num_objects': 2184, 'num_object_clones': 0, 'num_object_copies': 8736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9240, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 200}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:17.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:14:17.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9871", 'reported_seq': 9979, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.932261+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:14:14.932261+0000', 'last_peered': '2024-09-15T03:14:14.932261+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:14:14.932261+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9871, 'log_dups_size': 0, 'ondisk_log_size': 9871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19241809, 'num_objects': 2350, 'num_object_clones': 0, 'num_object_copies': 9400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2350, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9870, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 259}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:17.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:14:17.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9759", 'reported_seq': 9865, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.848379+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:14:14.848379+0000', 'last_peered': '2024-09-15T03:14:14.848379+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:14:14.848379+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9759, 'log_dups_size': 0, 'ondisk_log_size': 9759, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19019378, 'num_objects': 2322, 'num_object_clones': 0, 'num_object_copies': 9288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9757, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 238}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9792", 'reported_seq': 9876, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.747740+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:14:14.747740+0000', 'last_peered': '2024-09-15T03:14:14.747740+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:14:14.747740+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9792, 'log_dups_size': 0, 'ondisk_log_size': 9792, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18071552, 'num_objects': 2206, 'num_object_clones': 0, 'num_object_copies': 8824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9792, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 168}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9697", 'reported_seq': 9818, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.702972+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:14:14.702972+0000', 'last_peered': '2024-09-15T03:14:14.702972+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:14:14.702972+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9697, 'log_dups_size': 0, 'ondisk_log_size': 9697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18519666, 'num_objects': 2261, 'num_object_clones': 0, 'num_object_copies': 9044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9695, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 235}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9791", 'reported_seq': 9908, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.715613+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:14:14.715613+0000', 'last_peered': '2024-09-15T03:14:14.715613+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:14:14.715613+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9791, 'log_dups_size': 0, 'ondisk_log_size': 9791, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18790002, 'num_objects': 2294, 'num_object_clones': 0, 'num_object_copies': 9176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9789, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9649", 'reported_seq': 9740, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.791289+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:14:14.791289+0000', 'last_peered': '2024-09-15T03:14:14.791289+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:14:14.791289+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9649, 'log_dups_size': 0, 'ondisk_log_size': 9649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19008740, 'num_objects': 2321, 'num_object_clones': 0, 'num_object_copies': 9284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9646, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:14:17.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9685", 'reported_seq': 9804, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.811068+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:14:14.811068+0000', 'last_peered': '2024-09-15T03:14:14.811068+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:14:14.811068+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9685, 'log_dups_size': 0, 'ondisk_log_size': 9685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18699890, 'num_objects': 2283, 'num_object_clones': 0, 'num_object_copies': 9132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2283, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9683, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 241}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9684", 'reported_seq': 9778, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.726284+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:14:14.726284+0000', 'last_peered': '2024-09-15T03:14:14.726284+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:14:14.726284+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9684, 'log_dups_size': 0, 'ondisk_log_size': 9684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18010539, 'num_objects': 2199, 'num_object_clones': 0, 'num_object_copies': 8796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9681, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 137}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9755", 'reported_seq': 12775, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.950885+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:14:14.950885+0000', 'last_peered': '2024-09-15T03:14:14.950885+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:14:14.950885+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9755, 'log_dups_size': 0, 'ondisk_log_size': 9755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19479353, 'num_objects': 2378, 'num_object_clones': 0, 'num_object_copies': 9512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 571, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9754, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1470, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1807}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9472", 'reported_seq': 9572, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.870440+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:14:14.870440+0000', 'last_peered': '2024-09-15T03:14:14.870440+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:14:14.870440+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9472, 'log_dups_size': 0, 'ondisk_log_size': 9472, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18101874, 'num_objects': 2210, 'num_object_clones': 0, 'num_object_copies': 8840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9470, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 197}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9636", 'reported_seq': 9741, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.783013+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:14:14.783013+0000', 'last_peered': '2024-09-15T03:14:14.783013+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:14:14.783013+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9636, 'log_dups_size': 0, 'ondisk_log_size': 9636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18404978, 'num_objects': 2247, 'num_object_clones': 0, 'num_object_copies': 8988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9634, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 270}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:14:17.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9715", 'reported_seq': 9828, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:14.536659+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:14:14.536659+0000', 'last_peered': '2024-09-15T03:14:14.536659+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:14:14.536659+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9715, 'log_dups_size': 0, 'ondisk_log_size': 9715, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18790002, 'num_objects': 2294, 'num_object_clones': 0, 'num_object_copies': 9176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9713, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 235}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:17.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:14:19.976 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:19.977 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27019/145636 objects degraded (18.552%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":583,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36409,"data_bytes":298220116,"bytes_used":2754965504,"bytes_avail":383792091136,"bytes_total":386547056640,"degraded_objects":27019,"degraded_total":145636,"degraded_ratio":0.18552418358098272,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":155},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:14:20.085 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:14:20.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:21.716 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:21.732 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:21.789 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-15T03:14:20.894224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":292035156,"num_objects":35654,"num_object_clones":0,"num_object_copies":142616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155053,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5754,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155080,"ondisk_log_size":155080,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2659168,"kb_used_data":574016,"kb_used_omap":14,"kb_used_meta":2085105,"kb_avail":374828192,"statfs":{"total":386547056640,"available":383824068608,"internally_reserved":0,"allocated":587792384,"data_stored":587093666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135147547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,12,4,4,8,10,1,25],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21495808,"num_objects":-2624,"num_object_clones":0,"num_object_copies":-10496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005327"},"pg_stats":[{"pgid":"1.f","version":"35'9648","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.732641+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:19.732641+0000","last_peered":"2024-09-15T03:14:19.732641+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:19.732641+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9648,"log_dups_size":0,"ondisk_log_size":9648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18530304,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9648,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9286","reported_seq":9380,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.625042+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:19.625042+0000","last_peered":"2024-09-15T03:14:19.625042+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:19.625042+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9286,"log_dups_size":0,"ondisk_log_size":9286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17528434,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9284,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":156}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9915","reported_seq":10023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.699897+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:19.699897+0000","last_peered":"2024-09-15T03:14:19.699897+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:19.699897+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18881361,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9914,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9348","reported_seq":13320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:16.681567+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:16.681567+0000","last_peered":"2024-09-15T03:14:16.681567+0000","last_clean":"2024-09-15T03:14:16.681567+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:16.681567+0000","last_undegraded":"2024-09-15T03:14:16.681567+0000","last_fullsized":"2024-09-15T03:14:16.681567+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9348,"log_dups_size":0,"ondisk_log_size":9348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18675314,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9814","reported_seq":9920,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.943038+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:19.943038+0000","last_peered":"2024-09-15T03:14:19.943038+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:19.943038+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9814,"log_dups_size":0,"ondisk_log_size":9814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18568818,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9838","reported_seq":9922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.527335+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:19.527335+0000","last_peered":"2024-09-15T03:14:19.527335+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:19.527335+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17694720,"num_objects":2160,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9748","reported_seq":9869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.812300+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:19.812300+0000","last_peered":"2024-09-15T03:14:19.812300+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:19.812300+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18101874,"num_objects":2210,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9746,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":184}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9855","reported_seq":9972,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.828722+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:19.828722+0000","last_peered":"2024-09-15T03:14:19.828722+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:19.828722+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9855,"log_dups_size":0,"ondisk_log_size":9855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18265714,"num_objects":2230,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9853,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9707","reported_seq":9798,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.871677+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:19.871677+0000","last_peered":"2024-09-15T03:14:19.871677+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:19.871677+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18533604,"num_objects":2263,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9742","reported_seq":9861,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.819241+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:19.819241+0000","last_peered":"2024-09-15T03:14:19.819241+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:19.819241+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18232946,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9740,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":184}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9739","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.778564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:19.778564+0000","last_peered":"2024-09-15T03:14:19.778564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:19.778564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9739,"log_dups_size":0,"ondisk_log_size":9739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17559979,"num_objects":2144,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9736,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":82}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9635","reported_seq":13676,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:16.527577+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:16.527577+0000","last_peered":"2024-09-15T03:14:16.527577+0000","last_clean":"2024-09-15T03:14:16.527577+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:16.527577+0000","last_undegraded":"2024-09-15T03:14:16.527577+0000","last_fullsized":"2024-09-15T03:14:16.527577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9635,"log_dups_size":0,"ondisk_log_size":9635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18545465,"num_objects":2264,"num_object_clones":0,"num_object_copies":9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9634,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9813","reported_seq":13010,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:20.009674+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:20.009674+0000","last_peered":"2024-09-15T03:14:20.009674+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:20.009674+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19004217,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1559,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1838}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9529","reported_seq":9629,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.754863+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:19.754863+0000","last_peered":"2024-09-15T03:14:19.754863+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:19.754863+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17634930,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9686","reported_seq":9793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.220914+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:19.220914+0000","last_peered":"2024-09-15T03:14:19.220914+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:19.220914+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17995378,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9777","reported_seq":9890,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.829613+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:19.829613+0000","last_peered":"2024-09-15T03:14:19.829613+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:19.829613+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9777,"log_dups_size":0,"ondisk_log_size":9777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18282098,"num_objects":2232,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9775,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":173}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":292035156,"num_objects":35654,"num_object_clones":0,"num_object_copies":142616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35654,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155053,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5754,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585007104,"data_stored":585007104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155080,"ondisk_log_size":155080,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741616,"kb_used_data":142888,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93630224,"statfs":{"total":96636764160,"available":95877349376,"internally_reserved":0,"allocated":146317312,"data_stored":146141846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,25],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":20.984000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":8.0410000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":18.43},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":21.027999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":18.466000000000001},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":21.062000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728600,"kb_used_data":142864,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93643240,"statfs":{"total":96636764160,"available":95890677760,"internally_reserved":0,"allocated":146292736,"data_stored":146120416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.28999999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":6.0309999999999997},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":6.0629999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.55800000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455592,"kb_used_data":145248,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93916248,"statfs":{"total":96636764160,"available":96170237952,"internally_reserved":0,"allocated":148733952,"data_stored":148558486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,7,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.224}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.90300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.44900000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733360,"kb_used_data":143016,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93638480,"statfs":{"total":96636764160,"available":95885803520,"internally_reserved":0,"allocated":146448384,"data_stored":146272918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,8,1,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":3.4430000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":3.399},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.32900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":1.4630000000000001},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":16.562000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":145702912,"data_stored":145702912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148090880,"data_stored":148090880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145580032,"data_stored":145580032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145633280,"data_stored":145633280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:24.792 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:26.097 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:26.121 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:26.147 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-15T03:14:24.896408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289167956,"num_objects":35304,"num_object_clones":0,"num_object_copies":141216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155403,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5754,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155430,"ondisk_log_size":155430,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2643760,"kb_used_data":566892,"kb_used_omap":14,"kb_used_meta":2076785,"kb_avail":374843600,"statfs":{"total":386547056640,"available":383839846400,"internally_reserved":0,"allocated":580497408,"data_stored":579798690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126627867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,7,6,10,2,3,1,25],"upper_bound":1024},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16670720,"num_objects":-2035,"num_object_clones":0,"num_object_copies":-8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006565"},"pg_stats":[{"pgid":"1.f","version":"35'9648","reported_seq":9714,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.732641+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:19.732641+0000","last_peered":"2024-09-15T03:14:19.732641+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:19.732641+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9648,"log_dups_size":0,"ondisk_log_size":9648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18530304,"num_objects":2262,"num_object_clones":0,"num_object_copies":9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9648,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9342","reported_seq":9436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.567225+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:24.567225+0000","last_peered":"2024-09-15T03:14:24.567225+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:24.567225+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9342,"log_dups_size":0,"ondisk_log_size":9342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17069682,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9340,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9915","reported_seq":10023,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.699897+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:19.699897+0000","last_peered":"2024-09-15T03:14:19.699897+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:19.699897+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9915,"log_dups_size":0,"ondisk_log_size":9915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18881361,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9914,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":215}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9423","reported_seq":13395,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:21.833858+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:21.833858+0000","last_peered":"2024-09-15T03:14:21.833858+0000","last_clean":"2024-09-15T03:14:21.833858+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:21.833858+0000","last_undegraded":"2024-09-15T03:14:21.833858+0000","last_fullsized":"2024-09-15T03:14:21.833858+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18060914,"num_objects":2205,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9421,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9814","reported_seq":9920,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.943038+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:19.943038+0000","last_peered":"2024-09-15T03:14:19.943038+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:19.943038+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9814,"log_dups_size":0,"ondisk_log_size":9814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18568818,"num_objects":2267,"num_object_clones":0,"num_object_copies":9068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9838","reported_seq":9922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.527335+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:19.527335+0000","last_peered":"2024-09-15T03:14:19.527335+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:19.527335+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9838,"log_dups_size":0,"ondisk_log_size":9838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17694720,"num_objects":2160,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9838,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9805","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.858112+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:24.858112+0000","last_peered":"2024-09-15T03:14:24.858112+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:24.858112+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9805,"log_dups_size":0,"ondisk_log_size":9805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17634930,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9803,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":127}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9909","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.698534+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:24.698534+0000","last_peered":"2024-09-15T03:14:24.698534+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:24.698534+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17823346,"num_objects":2176,"num_object_clones":0,"num_object_copies":8704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":120}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9707","reported_seq":9798,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.871677+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:19.871677+0000","last_peered":"2024-09-15T03:14:19.871677+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:19.871677+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9707,"log_dups_size":0,"ondisk_log_size":9707,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18533604,"num_objects":2263,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9742","reported_seq":9861,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.819241+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:19.819241+0000","last_peered":"2024-09-15T03:14:19.819241+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:19.819241+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18232946,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9740,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":184}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9739","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.778564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:19.778564+0000","last_peered":"2024-09-15T03:14:19.778564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:19.778564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9739,"log_dups_size":0,"ondisk_log_size":9739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17559979,"num_objects":2144,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9736,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":82}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9685","reported_seq":13726,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:21.793711+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:21.793711+0000","last_peered":"2024-09-15T03:14:21.793711+0000","last_clean":"2024-09-15T03:14:21.793711+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:21.793711+0000","last_undegraded":"2024-09-15T03:14:21.793711+0000","last_fullsized":"2024-09-15T03:14:21.793711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18135865,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9813","reported_seq":13010,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:20.009674+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:20.009674+0000","last_peered":"2024-09-15T03:14:20.009674+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:20.009674+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19004217,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":482,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1559,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1838}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9529","reported_seq":9629,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.754863+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:19.754863+0000","last_peered":"2024-09-15T03:14:19.754863+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:19.754863+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9529,"log_dups_size":0,"ondisk_log_size":9529,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17634930,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9527,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":140}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9686","reported_seq":9793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:19.220914+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:19.220914+0000","last_peered":"2024-09-15T03:14:19.220914+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:19.220914+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17995378,"num_objects":2197,"num_object_clones":0,"num_object_copies":8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":220}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9835","reported_seq":9948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.811132+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:24.811132+0000","last_peered":"2024-09-15T03:14:24.811132+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:24.811132+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9835,"log_dups_size":0,"ondisk_log_size":9835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17806962,"num_objects":2174,"num_object_clones":0,"num_object_copies":8696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9833,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":115}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":289167956,"num_objects":35304,"num_object_clones":0,"num_object_copies":141216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26930,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35304,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155403,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5754,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577601536,"data_stored":577601536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155430,"ondisk_log_size":155430,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741616,"kb_used_data":142888,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93630224,"statfs":{"total":96636764160,"available":95877349376,"internally_reserved":0,"allocated":146317312,"data_stored":146141846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1,25],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":20.984000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":8.0410000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:13:18 2024","interfaces":[{"interface":"back","average":{"1min":3.5499999999999998,"5min":2.9900000000000002,"15min":2.0920000000000001},"min":{"1min":0.27600000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":22.440999999999999,"5min":40.710999999999999,"15min":40.710999999999999},"last":18.43},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.9750000000000001,"15min":1.4259999999999999},"min":{"1min":0.29399999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":22.262,"5min":22.262,"15min":22.262},"last":21.027999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":18.466000000000001},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":21.062000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728600,"kb_used_data":142864,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93643240,"statfs":{"total":96636764160,"available":95890677760,"internally_reserved":0,"allocated":146292736,"data_stored":146120416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.28999999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":6.0309999999999997},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":6.0629999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.55800000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439636,"kb_used_data":141760,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93932204,"statfs":{"total":96636764160,"available":96186576896,"internally_reserved":0,"allocated":145162240,"data_stored":144986774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.309}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.63500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.41399999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733908,"kb_used_data":139380,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93637932,"statfs":{"total":96636764160,"available":95885242368,"internally_reserved":0,"allocated":142725120,"data_stored":142549654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":2.7599999999999998},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.379},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.28799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":4.1289999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141938688,"data_stored":141938688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144449536,"data_stored":144449536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145580032,"data_stored":145580032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":145633280,"data_stored":145633280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:26.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:28.097 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:28.107 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:28.156 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-15T03:14:26.896879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":284383828,"num_objects":34720,"num_object_clones":0,"num_object_copies":138880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155987,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5810,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156014,"ondisk_log_size":156014,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2628164,"kb_used_data":559604,"kb_used_omap":14,"kb_used_meta":2068465,"kb_avail":374859196,"statfs":{"total":386547056640,"available":383855816704,"internally_reserved":0,"allocated":573034496,"data_stored":572335778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2118108187},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,9,10,15,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19546112,"num_objects":-2386,"num_object_clones":0,"num_object_copies":-9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":199,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006570"},"pg_stats":[{"pgid":"1.f","version":"35'9716","reported_seq":9782,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.758344+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:24.758344+0000","last_peered":"2024-09-15T03:14:24.758344+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:24.758344+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17973248,"num_objects":2194,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9716,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":228}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9342","reported_seq":9436,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.567225+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:24.567225+0000","last_peered":"2024-09-15T03:14:24.567225+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:24.567225+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9342,"log_dups_size":0,"ondisk_log_size":9342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17069682,"num_objects":2084,"num_object_clones":0,"num_object_copies":8336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9340,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9975","reported_seq":10083,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.992767+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:24.992767+0000","last_peered":"2024-09-15T03:14:24.992767+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:24.992767+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9975,"log_dups_size":0,"ondisk_log_size":9975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18389841,"num_objects":2246,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9974,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":155}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9423","reported_seq":13395,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:21.833858+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:21.833858+0000","last_peered":"2024-09-15T03:14:21.833858+0000","last_clean":"2024-09-15T03:14:21.833858+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:21.833858+0000","last_undegraded":"2024-09-15T03:14:21.833858+0000","last_fullsized":"2024-09-15T03:14:21.833858+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18060914,"num_objects":2205,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9421,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9882","reported_seq":9988,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.867558+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:24.867558+0000","last_peered":"2024-09-15T03:14:24.867558+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:24.867558+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9882,"log_dups_size":0,"ondisk_log_size":9882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18011762,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9880,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9895","reported_seq":9979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.546305+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:24.546305+0000","last_peered":"2024-09-15T03:14:24.546305+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:24.546305+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9895,"log_dups_size":0,"ondisk_log_size":9895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17227776,"num_objects":2103,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":65}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9805","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.858112+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:24.858112+0000","last_peered":"2024-09-15T03:14:24.858112+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:24.858112+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9805,"log_dups_size":0,"ondisk_log_size":9805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17634930,"num_objects":2153,"num_object_clones":0,"num_object_copies":8612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9803,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":127}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9909","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.698534+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:24.698534+0000","last_peered":"2024-09-15T03:14:24.698534+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:24.698534+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17823346,"num_objects":2176,"num_object_clones":0,"num_object_copies":8704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":120}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9765","reported_seq":9856,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.834033+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:24.834033+0000","last_peered":"2024-09-15T03:14:24.834033+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:24.834033+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18058468,"num_objects":2205,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9762,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9794","reported_seq":9913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.811673+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:24.811673+0000","last_peered":"2024-09-15T03:14:24.811673+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:24.811673+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9794,"log_dups_size":0,"ondisk_log_size":9794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17806962,"num_objects":2174,"num_object_clones":0,"num_object_copies":8696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":132}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9793","reported_seq":9887,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.880505+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:24.880505+0000","last_peered":"2024-09-15T03:14:24.880505+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:24.880505+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9793,"log_dups_size":0,"ondisk_log_size":9793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17117611,"num_objects":2090,"num_object_clones":0,"num_object_copies":8360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":28}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9685","reported_seq":13726,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:21.793711+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:21.793711+0000","last_peered":"2024-09-15T03:14:21.793711+0000","last_clean":"2024-09-15T03:14:21.793711+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:21.793711+0000","last_undegraded":"2024-09-15T03:14:21.793711+0000","last_fullsized":"2024-09-15T03:14:21.793711+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18135865,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9870","reported_seq":13180,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:25.056708+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:25.056708+0000","last_peered":"2024-09-15T03:14:25.056708+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:25.056708+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9870,"log_dups_size":0,"ondisk_log_size":9870,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18537273,"num_objects":2263,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":426,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9869,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1615,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1837}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9583","reported_seq":9683,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.844263+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:24.844263+0000","last_peered":"2024-09-15T03:14:24.844263+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:24.844263+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9583,"log_dups_size":0,"ondisk_log_size":9583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17192562,"num_objects":2099,"num_object_clones":0,"num_object_copies":8396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9581,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9742","reported_seq":9848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.892480+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:24.892480+0000","last_peered":"2024-09-15T03:14:24.892480+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:24.892480+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9742,"log_dups_size":0,"ondisk_log_size":9742,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17536626,"num_objects":2141,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9740,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9835","reported_seq":9948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:24.811132+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:24.811132+0000","last_peered":"2024-09-15T03:14:24.811132+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:24.811132+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9835,"log_dups_size":0,"ondisk_log_size":9835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17806962,"num_objects":2174,"num_object_clones":0,"num_object_copies":8696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9833,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":115}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":284383828,"num_objects":34720,"num_object_clones":0,"num_object_copies":138880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155987,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5810,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":570064896,"data_stored":570064896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156014,"ondisk_log_size":156014,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725488,"kb_used_data":139228,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93646352,"statfs":{"total":96636764160,"available":95893864448,"internally_reserved":0,"allocated":142569472,"data_stored":142394006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,3,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.41599999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.55500000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.25900000000000001},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729132,"kb_used_data":139236,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93642708,"statfs":{"total":96636764160,"available":95890132992,"internally_reserved":0,"allocated":142577664,"data_stored":142405344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.17399999999999999},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.20699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.42699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.24299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":439636,"kb_used_data":141760,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93932204,"statfs":{"total":96636764160,"available":96186576896,"internally_reserved":0,"allocated":145162240,"data_stored":144986774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,5,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.309}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.63500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.41399999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733908,"kb_used_data":139380,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93637932,"statfs":{"total":96636764160,"available":95885242368,"internally_reserved":0,"allocated":142725120,"data_stored":142549654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,4,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:13:22 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.9350000000000001,"15min":1.2370000000000001},"min":{"1min":0.19600000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":10.843,"5min":18.594000000000001,"15min":19.527000000000001},"last":2.7599999999999998},{"interface":"front","average":{"1min":2.895,"5min":3.3879999999999999,"15min":2.141},"min":{"1min":0.29399999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":29.763000000000002,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.41499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.379},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.28799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":4.1289999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":141938688,"data_stored":141938688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144449536,"data_stored":144449536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":141799424,"data_stored":141799424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":141877248,"data_stored":141877248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:28.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:14:28.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9716", 'reported_seq': 9782, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.758344+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:14:24.758344+0000', 'last_peered': '2024-09-15T03:14:24.758344+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:14:24.758344+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9716, 'log_dups_size': 0, 'ondisk_log_size': 9716, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17973248, 'num_objects': 2194, 'num_object_clones': 0, 'num_object_copies': 8776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9716, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 228}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:28.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:14:28.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9342", 'reported_seq': 9436, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.567225+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:14:24.567225+0000', 'last_peered': '2024-09-15T03:14:24.567225+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:14:24.567225+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9342, 'log_dups_size': 0, 'ondisk_log_size': 9342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17069682, 'num_objects': 2084, 'num_object_clones': 0, 'num_object_copies': 8336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9340, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:28.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:14:28.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9975", 'reported_seq': 10083, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.992767+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:14:24.992767+0000', 'last_peered': '2024-09-15T03:14:24.992767+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:14:24.992767+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9975, 'log_dups_size': 0, 'ondisk_log_size': 9975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18389841, 'num_objects': 2246, 'num_object_clones': 0, 'num_object_copies': 8984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2246, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9974, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 155}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:28.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9882", 'reported_seq': 9988, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.867558+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:14:24.867558+0000', 'last_peered': '2024-09-15T03:14:24.867558+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:14:24.867558+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9882, 'log_dups_size': 0, 'ondisk_log_size': 9882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18011762, 'num_objects': 2199, 'num_object_clones': 0, 'num_object_copies': 8796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9880, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 115}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9895", 'reported_seq': 9979, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.546305+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:14:24.546305+0000', 'last_peered': '2024-09-15T03:14:24.546305+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:14:24.546305+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9895, 'log_dups_size': 0, 'ondisk_log_size': 9895, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17227776, 'num_objects': 2103, 'num_object_clones': 0, 'num_object_copies': 8412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9895, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 65}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9805", 'reported_seq': 9925, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.858112+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:14:24.858112+0000', 'last_peered': '2024-09-15T03:14:24.858112+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:14:24.858112+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9805, 'log_dups_size': 0, 'ondisk_log_size': 9805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17634930, 'num_objects': 2153, 'num_object_clones': 0, 'num_object_copies': 8612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9803, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 127}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9909", 'reported_seq': 10026, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.698534+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:14:24.698534+0000', 'last_peered': '2024-09-15T03:14:24.698534+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:14:24.698534+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9909, 'log_dups_size': 0, 'ondisk_log_size': 9909, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17823346, 'num_objects': 2176, 'num_object_clones': 0, 'num_object_copies': 8704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9907, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 120}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9765", 'reported_seq': 9856, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.834033+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:14:24.834033+0000', 'last_peered': '2024-09-15T03:14:24.834033+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:14:24.834033+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9765, 'log_dups_size': 0, 'ondisk_log_size': 9765, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18058468, 'num_objects': 2205, 'num_object_clones': 0, 'num_object_copies': 8820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9762, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 178}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:14:28.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9794", 'reported_seq': 9913, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.811673+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:14:24.811673+0000', 'last_peered': '2024-09-15T03:14:24.811673+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:14:24.811673+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9794, 'log_dups_size': 0, 'ondisk_log_size': 9794, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17806962, 'num_objects': 2174, 'num_object_clones': 0, 'num_object_copies': 8696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9792, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 132}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9793", 'reported_seq': 9887, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.880505+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:14:24.880505+0000', 'last_peered': '2024-09-15T03:14:24.880505+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:14:24.880505+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9793, 'log_dups_size': 0, 'ondisk_log_size': 9793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17117611, 'num_objects': 2090, 'num_object_clones': 0, 'num_object_copies': 8360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9790, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 28}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9870", 'reported_seq': 13180, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:25.056708+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:14:25.056708+0000', 'last_peered': '2024-09-15T03:14:25.056708+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:14:25.056708+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9870, 'log_dups_size': 0, 'ondisk_log_size': 9870, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18537273, 'num_objects': 2263, 'num_object_clones': 0, 'num_object_copies': 9052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 426, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9869, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1615, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1837}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9583", 'reported_seq': 9683, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.844263+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:14:24.844263+0000', 'last_peered': '2024-09-15T03:14:24.844263+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:14:24.844263+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9583, 'log_dups_size': 0, 'ondisk_log_size': 9583, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17192562, 'num_objects': 2099, 'num_object_clones': 0, 'num_object_copies': 8396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9581, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 86}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9742", 'reported_seq': 9848, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.892480+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:14:24.892480+0000', 'last_peered': '2024-09-15T03:14:24.892480+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:14:24.892480+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9742, 'log_dups_size': 0, 'ondisk_log_size': 9742, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17536626, 'num_objects': 2141, 'num_object_clones': 0, 'num_object_copies': 8564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9740, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:14:28.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9835", 'reported_seq': 9948, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:24.811132+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:14:24.811132+0000', 'last_peered': '2024-09-15T03:14:24.811132+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:14:24.811132+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9835, 'log_dups_size': 0, 'ondisk_log_size': 9835, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17806962, 'num_objects': 2174, 'num_object_clones': 0, 'num_object_copies': 8696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9833, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 115}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:28.160 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:14:30.272 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:30.272 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26874/138452 objects degraded (19.410%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":593,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":34613,"data_bytes":283507284,"bytes_used":2696097792,"bytes_avail":383850958848,"bytes_total":386547056640,"degraded_objects":26874,"degraded_total":138452,"degraded_ratio":0.19410337156559673,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":145,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":149},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:14:30.349 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:14:30.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:31.988 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:32.088 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:32.129 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-15T03:14:30.899873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":276994644,"num_objects":33818,"num_object_clones":0,"num_object_copies":135272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33818,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156889,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5885,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156916,"ondisk_log_size":156916,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2634608,"kb_used_data":545296,"kb_used_omap":14,"kb_used_meta":2089265,"kb_avail":374852752,"statfs":{"total":386547056640,"available":383849218048,"internally_reserved":0,"allocated":558383104,"data_stored":557684386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139407387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,5,16,8,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21225472,"num_objects":-2591,"num_object_clones":0,"num_object_copies":-10364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2591,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008540"},"pg_stats":[{"pgid":"1.f","version":"35'9778","reported_seq":9844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.870953+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:29.870953+0000","last_peered":"2024-09-15T03:14:29.870953+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:29.870953+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9778,"log_dups_size":0,"ondisk_log_size":9778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17465344,"num_objects":2132,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9778,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9408","reported_seq":9502,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.948164+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:29.948164+0000","last_peered":"2024-09-15T03:14:29.948164+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:29.948164+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9408,"log_dups_size":0,"ondisk_log_size":9408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16529010,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9406,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":34}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10035","reported_seq":10143,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.671333+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:29.671333+0000","last_peered":"2024-09-15T03:14:29.671333+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:29.671333+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17898321,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10034,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":95}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9474","reported_seq":13445,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:26.872738+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:26.872738+0000","last_peered":"2024-09-15T03:14:26.872738+0000","last_clean":"2024-09-15T03:14:26.872738+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:26.872738+0000","last_undegraded":"2024-09-15T03:14:26.872738+0000","last_fullsized":"2024-09-15T03:14:26.872738+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9474,"log_dups_size":0,"ondisk_log_size":9474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17643122,"num_objects":2154,"num_object_clones":0,"num_object_copies":8616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9472,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'9937","reported_seq":10042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.924885+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:29.924885+0000","last_peered":"2024-09-15T03:14:29.924885+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:29.924885+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17561202,"num_objects":2144,"num_object_clones":0,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9935,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9942","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.985715+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:29.985715+0000","last_peered":"2024-09-15T03:14:29.985715+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:29.985715+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16842752,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9942,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":18}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9852","reported_seq":9973,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.693240+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:29.693240+0000","last_peered":"2024-09-15T03:14:29.693240+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:29.693240+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9852,"log_dups_size":0,"ondisk_log_size":9852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17249906,"num_objects":2106,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":80}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9954","reported_seq":10071,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.600872+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:29.600872+0000","last_peered":"2024-09-15T03:14:29.600872+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:29.600872+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9954,"log_dups_size":0,"ondisk_log_size":9954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454706,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9952,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":75}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9835","reported_seq":9926,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.984188+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:29.984188+0000","last_peered":"2024-09-15T03:14:29.984188+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:29.984188+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9835,"log_dups_size":0,"ondisk_log_size":9835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17485028,"num_objects":2135,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9832,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9843","reported_seq":9962,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.971370+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:29.971370+0000","last_peered":"2024-09-15T03:14:29.971370+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:29.971370+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9843,"log_dups_size":0,"ondisk_log_size":9843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17405554,"num_objects":2125,"num_object_clones":0,"num_object_copies":8500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9841,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":83}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9853","reported_seq":9947,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:30.022676+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:30.022676+0000","last_peered":"2024-09-15T03:14:30.022676+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:30.022676+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9853,"log_dups_size":0,"ondisk_log_size":9853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16626091,"num_objects":2030,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9850,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-32}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9741","reported_seq":13782,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:26.904778+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:26.904778+0000","last_peered":"2024-09-15T03:14:26.904778+0000","last_clean":"2024-09-15T03:14:26.904778+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:26.904778+0000","last_undegraded":"2024-09-15T03:14:26.904778+0000","last_fullsized":"2024-09-15T03:14:26.904778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9741,"log_dups_size":0,"ondisk_log_size":9741,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17677113,"num_objects":2158,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9740,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9916","reported_seq":13375,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:30.105827+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:30.105827+0000","last_peered":"2024-09-15T03:14:30.105827+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:30.105827+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9916,"log_dups_size":0,"ondisk_log_size":9916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18160441,"num_objects":2217,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9915,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1690,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9652","reported_seq":9752,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.923230+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:29.923230+0000","last_peered":"2024-09-15T03:14:29.923230+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:29.923230+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16627314,"num_objects":2030,"num_object_clones":0,"num_object_copies":8120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":17}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9806","reported_seq":9913,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.775701+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:29.775701+0000","last_peered":"2024-09-15T03:14:29.775701+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:29.775701+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9806,"log_dups_size":0,"ondisk_log_size":9806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17012338,"num_objects":2077,"num_object_clones":0,"num_object_copies":8308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9890","reported_seq":10003,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:29.896725+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:29.896725+0000","last_peered":"2024-09-15T03:14:29.896725+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:29.896725+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9890,"log_dups_size":0,"ondisk_log_size":9890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17356402,"num_objects":2119,"num_object_clones":0,"num_object_copies":8476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9888,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":60}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":276994644,"num_objects":33818,"num_object_clones":0,"num_object_copies":135272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26799,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33818,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156889,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5885,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":555069440,"data_stored":555069440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156916,"ondisk_log_size":156916,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725992,"kb_used_data":135584,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93645848,"statfs":{"total":96636764160,"available":95893348352,"internally_reserved":0,"allocated":138838016,"data_stored":138662550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,9,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.27700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.46700000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.67000000000000004},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.42999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729712,"kb_used_data":135656,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93642128,"statfs":{"total":96636764160,"available":95889539072,"internally_reserved":0,"allocated":138911744,"data_stored":138739424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":11.31},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.24299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.2610000000000001},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.2919999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.28199999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444380,"kb_used_data":138196,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93927460,"statfs":{"total":96636764160,"available":96181719040,"internally_reserved":0,"allocated":141512704,"data_stored":141337238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":35.463000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":25.661000000000001},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":13.446999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":34.875},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":33.444000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734524,"kb_used_data":135860,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93637316,"statfs":{"total":96636764160,"available":95884611584,"internally_reserved":0,"allocated":139120640,"data_stored":138945174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.46100000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.32100000000000001},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.40699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":138211328,"data_stored":138211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":140619776,"data_stored":140619776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":138104832,"data_stored":138104832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":138133504,"data_stored":138133504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:35.132 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:37.207 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:37.210 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:37.227 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-15T03:14:36.901428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":269441620,"num_objects":32896,"num_object_clones":0,"num_object_copies":131584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157811,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5950,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157838,"ondisk_log_size":157838,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2620172,"kb_used_data":530828,"kb_used_omap":14,"kb_used_meta":2089265,"kb_avail":374867188,"statfs":{"total":386547056640,"available":383864000512,"internally_reserved":0,"allocated":543567872,"data_stored":542869154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139407387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,13,9,9,10,12,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19726336,"num_objects":-2408,"num_object_clones":0,"num_object_copies":-9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2408,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005019"},"pg_stats":[{"pgid":"1.f","version":"35'9832","reported_seq":9896,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.034951+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:35.034951+0000","last_peered":"2024-09-15T03:14:35.034951+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:35.034951+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17022976,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9832,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9456","reported_seq":9550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.892429+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:34.892429+0000","last_peered":"2024-09-15T03:14:34.892429+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:34.892429+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9456,"log_dups_size":0,"ondisk_log_size":9456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16135794,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9454,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-14}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10100","reported_seq":10208,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.022011+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:35.022011+0000","last_peered":"2024-09-15T03:14:35.022011+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:35.022011+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17365841,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10099,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":30}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9524","reported_seq":13496,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:31.837683+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:31.837683+0000","last_peered":"2024-09-15T03:14:31.837683+0000","last_clean":"2024-09-15T03:14:31.837683+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:31.837683+0000","last_undegraded":"2024-09-15T03:14:31.837683+0000","last_fullsized":"2024-09-15T03:14:31.837683+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9524,"log_dups_size":0,"ondisk_log_size":9524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17233522,"num_objects":2104,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9522,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'10002","reported_seq":10108,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.804191+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:34.804191+0000","last_peered":"2024-09-15T03:14:34.804191+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:34.804191+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028722,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9992","reported_seq":10076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.011118+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:35.011118+0000","last_peered":"2024-09-15T03:14:35.011118+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:35.011118+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9992,"log_dups_size":0,"ondisk_log_size":9992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16433152,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-32}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9925","reported_seq":10046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.833775+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:34.833775+0000","last_peered":"2024-09-15T03:14:34.833775+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:34.833775+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16651890,"num_objects":2033,"num_object_clones":0,"num_object_copies":8132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9923,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10017","reported_seq":10134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.586122+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:34.586122+0000","last_peered":"2024-09-15T03:14:34.586122+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:34.586122+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16938610,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9885","reported_seq":9976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.730815+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:34.730815+0000","last_peered":"2024-09-15T03:14:34.730815+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:34.730815+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17075428,"num_objects":2085,"num_object_clones":0,"num_object_copies":8340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9882,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":58}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9897","reported_seq":10016,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.107090+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:35.107090+0000","last_peered":"2024-09-15T03:14:35.107090+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:35.107090+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16963186,"num_objects":2071,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9912","reported_seq":10006,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.997024+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:34.997024+0000","last_peered":"2024-09-15T03:14:34.997024+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:34.997024+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9912,"log_dups_size":0,"ondisk_log_size":9912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16142763,"num_objects":1971,"num_object_clones":0,"num_object_copies":7884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9909,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9792","reported_seq":13833,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:31.845696+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:31.845696+0000","last_peered":"2024-09-15T03:14:31.845696+0000","last_clean":"2024-09-15T03:14:31.845696+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:31.845696+0000","last_undegraded":"2024-09-15T03:14:31.845696+0000","last_fullsized":"2024-09-15T03:14:31.845696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9792,"log_dups_size":0,"ondisk_log_size":9792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17259321,"num_objects":2107,"num_object_clones":0,"num_object_copies":8428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9791,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9981","reported_seq":13571,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.107684+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:35.107684+0000","last_peered":"2024-09-15T03:14:35.107684+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:35.107684+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17627961,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9980,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1755,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9712","reported_seq":9812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.036507+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:35.036507+0000","last_peered":"2024-09-15T03:14:35.036507+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:35.036507+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9712,"log_dups_size":0,"ondisk_log_size":9712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16135794,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9710,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-43}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9859","reported_seq":9966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.032182+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:35.032182+0000","last_peered":"2024-09-15T03:14:35.032182+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:35.032182+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9859,"log_dups_size":0,"ondisk_log_size":9859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16578162,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9857,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9952","reported_seq":10065,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.908678+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:34.908678+0000","last_peered":"2024-09-15T03:14:34.908678+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:34.908678+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16848498,"num_objects":2057,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":269441620,"num_objects":32896,"num_object_clones":0,"num_object_copies":131584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32896,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157811,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5950,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540073984,"data_stored":540073984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157838,"ondisk_log_size":157838,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726532,"kb_used_data":131956,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93645308,"statfs":{"total":96636764160,"available":95892795392,"internally_reserved":0,"allocated":135122944,"data_stored":134947478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,5,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.24399999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.433}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.75900000000000001},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.45900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730324,"kb_used_data":132108,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93641516,"statfs":{"total":96636764160,"available":95888912384,"internally_reserved":0,"allocated":135278592,"data_stored":135106272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.192},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":1.0960000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":15.833},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":15.872999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":1.1910000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":444872,"kb_used_data":134528,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93926968,"statfs":{"total":96636764160,"available":96181215232,"internally_reserved":0,"allocated":137756672,"data_stored":137581206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,4,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.312},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":6.0430000000000001},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":4.9260000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.93999999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718444,"kb_used_data":132236,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93653396,"statfs":{"total":96636764160,"available":95901077504,"internally_reserved":0,"allocated":135409664,"data_stored":135234198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,10,2,2,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.22700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.193},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.26300000000000001},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134471680,"data_stored":134471680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136876032,"data_stored":136876032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134344704,"data_stored":134344704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":134381568,"data_stored":134381568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:37.228 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:39.047 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:39.070 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:39.083 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-15T03:14:38.901991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":268532308,"num_objects":32785,"num_object_clones":0,"num_object_copies":131140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157922,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5950,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157949,"ondisk_log_size":157949,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2604072,"kb_used_data":527208,"kb_used_omap":14,"kb_used_meta":2076785,"kb_avail":374883288,"statfs":{"total":386547056640,"available":383880486912,"internally_reserved":0,"allocated":539860992,"data_stored":539162274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126627867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,17,9,11,7,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15851520,"num_objects":-1935,"num_object_clones":0,"num_object_copies":-7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1935,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005112"},"pg_stats":[{"pgid":"1.f","version":"35'9832","reported_seq":9896,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.034951+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:35.034951+0000","last_peered":"2024-09-15T03:14:35.034951+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:35.034951+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9832,"log_dups_size":0,"ondisk_log_size":9832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17022976,"num_objects":2078,"num_object_clones":0,"num_object_copies":8312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9832,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9456","reported_seq":9550,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.892429+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:34.892429+0000","last_peered":"2024-09-15T03:14:34.892429+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:34.892429+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9456,"log_dups_size":0,"ondisk_log_size":9456,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16135794,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9454,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-14}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10100","reported_seq":10208,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.022011+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:35.022011+0000","last_peered":"2024-09-15T03:14:35.022011+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:35.022011+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17365841,"num_objects":2121,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10099,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":30}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9580","reported_seq":13552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:37.162946+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:37.162946+0000","last_peered":"2024-09-15T03:14:37.162946+0000","last_clean":"2024-09-15T03:14:37.162946+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:37.162946+0000","last_undegraded":"2024-09-15T03:14:37.162946+0000","last_fullsized":"2024-09-15T03:14:37.162946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16774770,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'10002","reported_seq":10108,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.804191+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:34.804191+0000","last_peered":"2024-09-15T03:14:34.804191+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:34.804191+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028722,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9992","reported_seq":10076,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.011118+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:35.011118+0000","last_peered":"2024-09-15T03:14:35.011118+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:35.011118+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9992,"log_dups_size":0,"ondisk_log_size":9992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16433152,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-32}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9925","reported_seq":10046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.833775+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:34.833775+0000","last_peered":"2024-09-15T03:14:34.833775+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:34.833775+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9925,"log_dups_size":0,"ondisk_log_size":9925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16651890,"num_objects":2033,"num_object_clones":0,"num_object_copies":8132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9923,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10017","reported_seq":10134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.586122+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:34.586122+0000","last_peered":"2024-09-15T03:14:34.586122+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:34.586122+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16938610,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10015,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9885","reported_seq":9976,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.730815+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:34.730815+0000","last_peered":"2024-09-15T03:14:34.730815+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:34.730815+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9885,"log_dups_size":0,"ondisk_log_size":9885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17075428,"num_objects":2085,"num_object_clones":0,"num_object_copies":8340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9882,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":58}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9897","reported_seq":10016,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.107090+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:35.107090+0000","last_peered":"2024-09-15T03:14:35.107090+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:35.107090+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16963186,"num_objects":2071,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9912","reported_seq":10006,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.997024+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:34.997024+0000","last_peered":"2024-09-15T03:14:34.997024+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:34.997024+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9912,"log_dups_size":0,"ondisk_log_size":9912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16142763,"num_objects":1971,"num_object_clones":0,"num_object_copies":7884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9909,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-91}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9847","reported_seq":13888,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:37.016758+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:37.016758+0000","last_peered":"2024-09-15T03:14:37.016758+0000","last_clean":"2024-09-15T03:14:37.016758+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:37.016758+0000","last_undegraded":"2024-09-15T03:14:37.016758+0000","last_fullsized":"2024-09-15T03:14:37.016758+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9847,"log_dups_size":0,"ondisk_log_size":9847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16808761,"num_objects":2052,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'9981","reported_seq":13571,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.107684+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:35.107684+0000","last_peered":"2024-09-15T03:14:35.107684+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:35.107684+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17627961,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9980,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1755,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1866}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9712","reported_seq":9812,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.036507+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:35.036507+0000","last_peered":"2024-09-15T03:14:35.036507+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:35.036507+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9712,"log_dups_size":0,"ondisk_log_size":9712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16135794,"num_objects":1970,"num_object_clones":0,"num_object_copies":7880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9710,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-43}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9859","reported_seq":9966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:35.032182+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:35.032182+0000","last_peered":"2024-09-15T03:14:35.032182+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:35.032182+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9859,"log_dups_size":0,"ondisk_log_size":9859,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16578162,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9857,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":47}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9952","reported_seq":10065,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:34.908678+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:34.908678+0000","last_peered":"2024-09-15T03:14:34.908678+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:34.908678+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9952,"log_dups_size":0,"ondisk_log_size":9952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16848498,"num_objects":2057,"num_object_clones":0,"num_object_copies":8228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":268532308,"num_objects":32785,"num_object_clones":0,"num_object_copies":131140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157922,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5950,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536236032,"data_stored":536236032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157949,"ondisk_log_size":157949,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726532,"kb_used_data":131956,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93645308,"statfs":{"total":96636764160,"available":95892795392,"internally_reserved":0,"allocated":135122944,"data_stored":134947478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,5,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.24399999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.433}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.75900000000000001},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.45900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730324,"kb_used_data":132108,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93641516,"statfs":{"total":96636764160,"available":95888912384,"internally_reserved":0,"allocated":135278592,"data_stored":135106272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.641,"15min":1.631},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.16300000000000001},"max":{"1min":11.372,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.192},{"interface":"front","average":{"1min":1.895,"5min":1.9530000000000001,"15min":1.3129999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":19.277000000000001,"5min":19.329000000000001,"15min":19.329000000000001},"last":1.0960000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":15.833},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":15.872999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:13:37 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":2.125,"15min":1.377},"min":{"1min":0.28000000000000003,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":6.75,"5min":15.675000000000001,"15min":21.600999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":2.1890000000000001,"5min":2.3759999999999999,"15min":1.524},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.23400000000000001},"max":{"1min":19.343,"5min":22.710000000000001,"15min":22.710000000000001},"last":1.1910000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428772,"kb_used_data":130908,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93943068,"statfs":{"total":96636764160,"available":96197701632,"internally_reserved":0,"allocated":134049792,"data_stored":133874326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,1,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":20.346},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.81399999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.254},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.31}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":6.6040000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.29499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718444,"kb_used_data":132236,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93653396,"statfs":{"total":96636764160,"available":95901077504,"internally_reserved":0,"allocated":135409664,"data_stored":135234198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,10,2,2,3,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.22700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":0.193},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.28999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.26300000000000001},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134471680,"data_stored":134471680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":133038080,"data_stored":133038080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":134344704,"data_stored":134344704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":134381568,"data_stored":134381568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:39.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:14:39.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9832", 'reported_seq': 9896, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:35.034951+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:14:35.034951+0000', 'last_peered': '2024-09-15T03:14:35.034951+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:14:35.034951+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9832, 'log_dups_size': 0, 'ondisk_log_size': 9832, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17022976, 'num_objects': 2078, 'num_object_clones': 0, 'num_object_copies': 8312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9832, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 112}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:39.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:14:39.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9456", 'reported_seq': 9550, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:34.892429+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:14:34.892429+0000', 'last_peered': '2024-09-15T03:14:34.892429+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:14:34.892429+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9456, 'log_dups_size': 0, 'ondisk_log_size': 9456, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16135794, 'num_objects': 1970, 'num_object_clones': 0, 'num_object_copies': 7880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9454, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -14}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10100", 'reported_seq': 10208, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:35.022011+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:14:35.022011+0000', 'last_peered': '2024-09-15T03:14:35.022011+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:14:35.022011+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17365841, 'num_objects': 2121, 'num_object_clones': 0, 'num_object_copies': 8484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2121, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10099, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 30}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10002", 'reported_seq': 10108, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:34.804191+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:14:34.804191+0000', 'last_peered': '2024-09-15T03:14:34.804191+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:14:34.804191+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17028722, 'num_objects': 2079, 'num_object_clones': 0, 'num_object_copies': 8316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10000, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -5}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9992", 'reported_seq': 10076, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:35.011118+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:14:35.011118+0000', 'last_peered': '2024-09-15T03:14:35.011118+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:14:35.011118+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9992, 'log_dups_size': 0, 'ondisk_log_size': 9992, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16433152, 'num_objects': 2006, 'num_object_clones': 0, 'num_object_copies': 8024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9992, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -32}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9925", 'reported_seq': 10046, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:34.833775+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:14:34.833775+0000', 'last_peered': '2024-09-15T03:14:34.833775+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:14:34.833775+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9925, 'log_dups_size': 0, 'ondisk_log_size': 9925, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16651890, 'num_objects': 2033, 'num_object_clones': 0, 'num_object_copies': 8132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2033, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9923, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 7}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:14:39.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10017", 'reported_seq': 10134, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:34.586122+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:14:34.586122+0000', 'last_peered': '2024-09-15T03:14:34.586122+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:14:34.586122+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16938610, 'num_objects': 2068, 'num_object_clones': 0, 'num_object_copies': 8272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10015, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 12}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9885", 'reported_seq': 9976, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:34.730815+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:14:34.730815+0000', 'last_peered': '2024-09-15T03:14:34.730815+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:14:34.730815+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9885, 'log_dups_size': 0, 'ondisk_log_size': 9885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17075428, 'num_objects': 2085, 'num_object_clones': 0, 'num_object_copies': 8340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9882, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 58}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9897", 'reported_seq': 10016, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:35.107090+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:14:35.107090+0000', 'last_peered': '2024-09-15T03:14:35.107090+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:14:35.107090+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9897, 'log_dups_size': 0, 'ondisk_log_size': 9897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16963186, 'num_objects': 2071, 'num_object_clones': 0, 'num_object_copies': 8284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9895, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 29}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'9912", 'reported_seq': 10006, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:34.997024+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:14:34.997024+0000', 'last_peered': '2024-09-15T03:14:34.997024+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:14:34.997024+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9912, 'log_dups_size': 0, 'ondisk_log_size': 9912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16142763, 'num_objects': 1971, 'num_object_clones': 0, 'num_object_copies': 7884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9909, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -91}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9981", 'reported_seq': 13571, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:35.107684+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:14:35.107684+0000', 'last_peered': '2024-09-15T03:14:35.107684+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:14:35.107684+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9981, 'log_dups_size': 0, 'ondisk_log_size': 9981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17627961, 'num_objects': 2152, 'num_object_clones': 0, 'num_object_copies': 8608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 286, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9980, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1755, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1866}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:14:39.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9712", 'reported_seq': 9812, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:35.036507+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:14:35.036507+0000', 'last_peered': '2024-09-15T03:14:35.036507+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:14:35.036507+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9712, 'log_dups_size': 0, 'ondisk_log_size': 9712, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16135794, 'num_objects': 1970, 'num_object_clones': 0, 'num_object_copies': 7880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9710, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -43}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:39.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:14:39.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9859", 'reported_seq': 9966, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:35.032182+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:14:35.032182+0000', 'last_peered': '2024-09-15T03:14:35.032182+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:14:35.032182+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9859, 'log_dups_size': 0, 'ondisk_log_size': 9859, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16578162, 'num_objects': 2024, 'num_object_clones': 0, 'num_object_copies': 8096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9857, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 47}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:39.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:14:39.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9952", 'reported_seq': 10065, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:34.908678+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:14:34.908678+0000', 'last_peered': '2024-09-15T03:14:34.908678+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:14:34.908678+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9952, 'log_dups_size': 0, 'ondisk_log_size': 9952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16848498, 'num_objects': 2057, 'num_object_clones': 0, 'num_object_copies': 8228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2057, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9950, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:39.087 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:14:40.896 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:40.897 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26734/131140 objects degraded (20.386%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":603,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32785,"data_bytes":268532308,"bytes_used":2666569728,"bytes_avail":383880486912,"bytes_total":386547056640,"degraded_objects":26734,"degraded_total":131140,"degraded_ratio":0.20385847186213207,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":140,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":161},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:14:40.986 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:14:40.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:42.672 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:42.691 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:42.728 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-09-15T03:14:40.902448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":262117972,"num_objects":32002,"num_object_clones":0,"num_object_copies":128008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158705,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6001,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158632,"ondisk_log_size":158632,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2593112,"kb_used_data":516280,"kb_used_omap":14,"kb_used_meta":2076785,"kb_avail":374894248,"statfs":{"total":386547056640,"available":383891709952,"internally_reserved":0,"allocated":528670720,"data_stored":527972002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126627867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,11,3,9,10,13,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21389312,"num_objects":-2611,"num_object_clones":0,"num_object_copies":-10444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2611,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005045"},"pg_stats":[{"pgid":"1.f","version":"35'9894","reported_seq":9960,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:39.891049+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:39.891049+0000","last_peered":"2024-09-15T03:14:39.891049+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:39.891049+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9894,"log_dups_size":0,"ondisk_log_size":9894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":2016,"num_object_clones":0,"num_object_copies":8064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9894,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":50}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9509","reported_seq":9603,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:39.917574+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:39.917574+0000","last_peered":"2024-09-15T03:14:39.917574+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:39.917574+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9509,"log_dups_size":0,"ondisk_log_size":9509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15701618,"num_objects":1917,"num_object_clones":0,"num_object_copies":7668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9507,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-67}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10146","reported_seq":10254,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:39.853271+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:39.853271+0000","last_peered":"2024-09-15T03:14:39.853271+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:39.853271+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16989009,"num_objects":2075,"num_object_clones":0,"num_object_copies":8300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10145,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-16}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9580","reported_seq":13552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:37.162946+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:37.162946+0000","last_peered":"2024-09-15T03:14:37.162946+0000","last_clean":"2024-09-15T03:14:37.162946+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:37.162946+0000","last_undegraded":"2024-09-15T03:14:37.162946+0000","last_fullsized":"2024-09-15T03:14:37.162946+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9580,"log_dups_size":0,"ondisk_log_size":9580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16774770,"num_objects":2048,"num_object_clones":0,"num_object_copies":8192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'10046","reported_seq":10152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:39.963369+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:39.963369+0000","last_peered":"2024-09-15T03:14:39.963369+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:39.963369+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16668274,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10037","reported_seq":10121,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:40.138101+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:40.138101+0000","last_peered":"2024-09-15T03:14:40.138101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:40.138101+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16064512,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-77}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9986","reported_seq":10107,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:39.752569+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:39.752569+0000","last_peered":"2024-09-15T03:14:39.752569+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:39.752569+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9986,"log_dups_size":0,"ondisk_log_size":9986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16152178,"num_objects":1972,"num_object_clones":0,"num_object_copies":7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9984,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-54}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10080","reported_seq":10197,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:39.949647+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:39.949647+0000","last_peered":"2024-09-15T03:14:39.949647+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:39.949647+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16422514,"num_objects":2005,"num_object_clones":0,"num_object_copies":8020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10078,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-51}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9938","reported_seq":10029,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:40.033055+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:40.033055+0000","last_peered":"2024-09-15T03:14:40.033055+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:40.033055+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9938,"log_dups_size":0,"ondisk_log_size":9938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16641252,"num_objects":2032,"num_object_clones":0,"num_object_copies":8128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9935,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9959","reported_seq":10077,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:40.071043+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:40.071043+0000","last_peered":"2024-09-15T03:14:40.071043+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:40.071043+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9959,"log_dups_size":0,"ondisk_log_size":9959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16455282,"num_objects":2009,"num_object_clones":0,"num_object_copies":8036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9957,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-33}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9958","reported_seq":10052,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:39.752829+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:39.752829+0000","last_peered":"2024-09-15T03:14:39.752829+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:39.752829+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9958,"log_dups_size":0,"ondisk_log_size":9958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15765931,"num_objects":1925,"num_object_clones":0,"num_object_copies":7700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9955,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-137}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9847","reported_seq":13888,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:37.016758+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:37.016758+0000","last_peered":"2024-09-15T03:14:37.016758+0000","last_clean":"2024-09-15T03:14:37.016758+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:37.016758+0000","last_undegraded":"2024-09-15T03:14:37.016758+0000","last_fullsized":"2024-09-15T03:14:37.016758+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9847,"log_dups_size":0,"ondisk_log_size":9847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16808761,"num_objects":2052,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9846,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10047","reported_seq":13739,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:40.186141+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:40.186141+0000","last_peered":"2024-09-15T03:14:40.186141+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:40.186141+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17087289,"num_objects":2086,"num_object_clones":0,"num_object_copies":8344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10046,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1806,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9773","reported_seq":9873,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:40.177896+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:40.177896+0000","last_peered":"2024-09-15T03:14:40.177896+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:40.177896+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15636082,"num_objects":1909,"num_object_clones":0,"num_object_copies":7636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9771,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9922","reported_seq":10029,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:40.072776+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:40.072776+0000","last_peered":"2024-09-15T03:14:40.072776+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:40.072776+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16062066,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9920,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-16}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10010","reported_seq":10123,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:40.004193+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:40.004193+0000","last_peered":"2024-09-15T03:14:40.004193+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:40.004193+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16373362,"num_objects":1999,"num_object_clones":0,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10008,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-60}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":262117972,"num_objects":32002,"num_object_clones":0,"num_object_copies":128008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26683,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158705,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6001,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":525201408,"data_stored":525201408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158632,"ondisk_log_size":158632,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727000,"kb_used_data":128272,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93644840,"statfs":{"total":96636764160,"available":95892316160,"internally_reserved":0,"allocated":131350528,"data_stored":131175062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,5,5,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":4.5960000000000001},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.247}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":4.6230000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":4.5540000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718356,"kb_used_data":128460,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93653484,"statfs":{"total":96636764160,"available":95901167616,"internally_reserved":0,"allocated":131543040,"data_stored":131370720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.251},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.20000000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":14.814},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":14.846}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.42099999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428772,"kb_used_data":130908,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93943068,"statfs":{"total":96636764160,"available":96197701632,"internally_reserved":0,"allocated":134049792,"data_stored":133874326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,1,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":20.346},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.81399999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.254},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.31}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":6.6040000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.29499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718984,"kb_used_data":128640,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93652856,"statfs":{"total":96636764160,"available":95900524544,"internally_reserved":0,"allocated":131727360,"data_stored":131551894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,4,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":9.7249999999999996},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":8.0060000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:13:40 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":2.8900000000000001,"15min":1.8460000000000001},"min":{"1min":0.254,"5min":0.216,"15min":0.16200000000000001},"max":{"1min":9.5909999999999993,"5min":30.306000000000001,"15min":30.802},"last":3.7130000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.375,"15min":1.3009999999999999},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":19.722000000000001,"5min":20.387,"15min":20.387},"last":0.25900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":3.2530000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130797568,"data_stored":130797568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":133038080,"data_stored":133038080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130691072,"data_stored":130691072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130674688,"data_stored":130674688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:45.734 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:47.509 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:47.522 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:47.595 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-15T03:14:46.907392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":254474836,"num_objects":31069,"num_object_clones":0,"num_object_copies":124276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159638,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6048,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159265,"ondisk_log_size":159265,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2576692,"kb_used_data":499808,"kb_used_omap":14,"kb_used_meta":2076785,"kb_avail":374910668,"statfs":{"total":386547056640,"available":383908524032,"internally_reserved":0,"allocated":511803392,"data_stored":511104674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126627867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,6,13,14,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21692416,"num_objects":-2648,"num_object_clones":0,"num_object_copies":-10592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006435"},"pg_stats":[{"pgid":"1.f","version":"35'9950","reported_seq":10016,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.122217+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:45.122217+0000","last_peered":"2024-09-15T03:14:45.122217+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:45.122217+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9950,"log_dups_size":0,"ondisk_log_size":9950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16056320,"num_objects":1960,"num_object_clones":0,"num_object_copies":7840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9588","reported_seq":9682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.035765+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:45.035765+0000","last_peered":"2024-09-15T03:14:45.035765+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:45.035765+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15054450,"num_objects":1838,"num_object_clones":0,"num_object_copies":7352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9586,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10210","reported_seq":10318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.260681+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:45.260681+0000","last_peered":"2024-09-15T03:14:45.260681+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:45.260681+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16464721,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10209,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-80}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9649","reported_seq":13620,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:42.261224+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:42.261224+0000","last_peered":"2024-09-15T03:14:42.261224+0000","last_clean":"2024-09-15T03:14:42.261224+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:42.261224+0000","last_undegraded":"2024-09-15T03:14:42.261224+0000","last_fullsized":"2024-09-15T03:14:42.261224+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16209522,"num_objects":1979,"num_object_clones":0,"num_object_copies":7916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'10098","reported_seq":10204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.021074+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:45.021074+0000","last_peered":"2024-09-15T03:14:45.021074+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:45.021074+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16242290,"num_objects":1983,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10087","reported_seq":10171,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.208814+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:45.208814+0000","last_peered":"2024-09-15T03:14:45.208814+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:45.208814+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15654912,"num_objects":1911,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10087,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10045","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.084585+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:45.084585+0000","last_peered":"2024-09-15T03:14:45.084585+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:45.084585+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15668850,"num_objects":1913,"num_object_clones":0,"num_object_copies":7652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-113}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10136","reported_seq":10253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:44.883022+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:44.883022+0000","last_peered":"2024-09-15T03:14:44.883022+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:44.883022+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15963762,"num_objects":1949,"num_object_clones":0,"num_object_copies":7796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10134,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9990","reported_seq":10081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:44.841887+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:44.841887+0000","last_peered":"2024-09-15T03:14:44.841887+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:44.841887+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16215268,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10014","reported_seq":10133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.256243+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:45.256243+0000","last_peered":"2024-09-15T03:14:45.256243+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:45.256243+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16004722,"num_objects":1954,"num_object_clones":0,"num_object_copies":7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10012,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-88}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10016","reported_seq":10110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.201489+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:45.201489+0000","last_peered":"2024-09-15T03:14:45.201489+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:45.201489+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15290795,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9904","reported_seq":13944,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:42.203401+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:42.203401+0000","last_peered":"2024-09-15T03:14:42.203401+0000","last_clean":"2024-09-15T03:14:42.203401+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:42.203401+0000","last_undegraded":"2024-09-15T03:14:42.203401+0000","last_fullsized":"2024-09-15T03:14:42.203401+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16341817,"num_objects":1995,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9903,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10120","reported_seq":13906,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.353495+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:45.353495+0000","last_peered":"2024-09-15T03:14:45.353495+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:45.353495+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16489273,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9824","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:44.550373+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:44.550373+0000","last_peered":"2024-09-15T03:14:44.550373+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:44.550373+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9824,"log_dups_size":0,"ondisk_log_size":9824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15218290,"num_objects":1858,"num_object_clones":0,"num_object_copies":7432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9822,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-155}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9978","reported_seq":10085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.246399+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:45.246399+0000","last_peered":"2024-09-15T03:14:45.246399+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:45.246399+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15603314,"num_objects":1905,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-72}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10056","reported_seq":10169,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.104371+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:45.104371+0000","last_peered":"2024-09-15T03:14:45.104371+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:45.104371+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15996530,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":254474836,"num_objects":31069,"num_object_clones":0,"num_object_copies":124276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159638,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6048,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":510025728,"data_stored":510025728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159265,"ondisk_log_size":159265,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711016,"kb_used_data":124764,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93660824,"statfs":{"total":96636764160,"available":95908683776,"internally_reserved":0,"allocated":127758336,"data_stored":127582870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,3,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":8.1820000000000004},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.22700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":9.7929999999999993},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.373}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718236,"kb_used_data":124140,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93653604,"statfs":{"total":96636764160,"available":95901290496,"internally_reserved":0,"allocated":127119360,"data_stored":126947040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.29099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":12.867000000000001},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":12.928000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.375},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.35099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428612,"kb_used_data":126588,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93943228,"statfs":{"total":96636764160,"available":96197865472,"internally_reserved":0,"allocated":129626112,"data_stored":129450646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.251},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":10.489000000000001},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":2.5009999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":4.3559999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":10.539}]}]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718828,"kb_used_data":124316,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93653012,"statfs":{"total":96636764160,"available":95900684288,"internally_reserved":0,"allocated":127299584,"data_stored":127124118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":5.7549999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.25900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":5.8280000000000003},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.81899999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":126976000,"data_stored":126976000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129323008,"data_stored":129323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126828544,"data_stored":126828544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":126898176,"data_stored":126898176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:47.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:49.596 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:49.688 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:49.732 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-15T03:14:48.907950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253491796,"num_objects":30949,"num_object_clones":0,"num_object_copies":123796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30949,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159758,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6048,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159385,"ondisk_log_size":159385,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2581412,"kb_used_data":496196,"kb_used_omap":14,"kb_used_meta":2085105,"kb_avail":374905948,"statfs":{"total":386547056640,"available":383903690752,"internally_reserved":0,"allocated":508104704,"data_stored":507405986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135147547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,8,4,13,14,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15949824,"num_objects":-1947,"num_object_clones":0,"num_object_copies":-7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1947,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006521"},"pg_stats":[{"pgid":"1.f","version":"35'9950","reported_seq":10016,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.122217+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:45.122217+0000","last_peered":"2024-09-15T03:14:45.122217+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:45.122217+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9950,"log_dups_size":0,"ondisk_log_size":9950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16056320,"num_objects":1960,"num_object_clones":0,"num_object_copies":7840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9950,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-6}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9588","reported_seq":9682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.035765+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:45.035765+0000","last_peered":"2024-09-15T03:14:45.035765+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:45.035765+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9588,"log_dups_size":0,"ondisk_log_size":9588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15054450,"num_objects":1838,"num_object_clones":0,"num_object_copies":7352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9586,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-146}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10210","reported_seq":10318,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.260681+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:45.260681+0000","last_peered":"2024-09-15T03:14:45.260681+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:45.260681+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16464721,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10209,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-80}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9705","reported_seq":13677,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:47.192393+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:47.192393+0000","last_peered":"2024-09-15T03:14:47.192393+0000","last_clean":"2024-09-15T03:14:47.192393+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:47.192393+0000","last_undegraded":"2024-09-15T03:14:47.192393+0000","last_fullsized":"2024-09-15T03:14:47.192393+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9705,"log_dups_size":0,"ondisk_log_size":9705,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15750770,"num_objects":1923,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9703,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'10098","reported_seq":10204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.021074+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:45.021074+0000","last_peered":"2024-09-15T03:14:45.021074+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:45.021074+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16242290,"num_objects":1983,"num_object_clones":0,"num_object_copies":7932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10096,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-101}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10087","reported_seq":10171,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.208814+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:45.208814+0000","last_peered":"2024-09-15T03:14:45.208814+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:45.208814+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15654912,"num_objects":1911,"num_object_clones":0,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10087,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10045","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.084585+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:45.084585+0000","last_peered":"2024-09-15T03:14:45.084585+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:45.084585+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15668850,"num_objects":1913,"num_object_clones":0,"num_object_copies":7652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10043,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-113}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10136","reported_seq":10253,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:44.883022+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:44.883022+0000","last_peered":"2024-09-15T03:14:44.883022+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:44.883022+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15963762,"num_objects":1949,"num_object_clones":0,"num_object_copies":7796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10134,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-107}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9990","reported_seq":10081,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:44.841887+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:44.841887+0000","last_peered":"2024-09-15T03:14:44.841887+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:44.841887+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16215268,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-47}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10014","reported_seq":10133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.256243+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:45.256243+0000","last_peered":"2024-09-15T03:14:45.256243+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:45.256243+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16004722,"num_objects":1954,"num_object_clones":0,"num_object_copies":7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10012,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-88}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10016","reported_seq":10110,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.201489+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:45.201489+0000","last_peered":"2024-09-15T03:14:45.201489+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:45.201489+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15290795,"num_objects":1867,"num_object_clones":0,"num_object_copies":7468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9968","reported_seq":14009,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:47.372886+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:47.372886+0000","last_peered":"2024-09-15T03:14:47.372886+0000","last_clean":"2024-09-15T03:14:47.372886+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:47.372886+0000","last_undegraded":"2024-09-15T03:14:47.372886+0000","last_fullsized":"2024-09-15T03:14:47.372886+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9968,"log_dups_size":0,"ondisk_log_size":9968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15817529,"num_objects":1931,"num_object_clones":0,"num_object_copies":7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9967,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10120","reported_seq":13906,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.353495+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:45.353495+0000","last_peered":"2024-09-15T03:14:45.353495+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:45.353495+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16489273,"num_objects":2013,"num_object_clones":0,"num_object_copies":8052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1853,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9824","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:44.550373+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:44.550373+0000","last_peered":"2024-09-15T03:14:44.550373+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:44.550373+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9824,"log_dups_size":0,"ondisk_log_size":9824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15218290,"num_objects":1858,"num_object_clones":0,"num_object_copies":7432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9822,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-155}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9978","reported_seq":10085,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.246399+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:45.246399+0000","last_peered":"2024-09-15T03:14:45.246399+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:45.246399+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9978,"log_dups_size":0,"ondisk_log_size":9978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15603314,"num_objects":1905,"num_object_clones":0,"num_object_copies":7620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9976,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-72}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10056","reported_seq":10169,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:45.104371+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:45.104371+0000","last_peered":"2024-09-15T03:14:45.104371+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:45.104371+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15996530,"num_objects":1953,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":253491796,"num_objects":30949,"num_object_clones":0,"num_object_copies":123796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30949,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159758,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6048,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":506187776,"data_stored":506187776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159385,"ondisk_log_size":159385,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711016,"kb_used_data":124764,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93660824,"statfs":{"total":96636764160,"available":95908683776,"internally_reserved":0,"allocated":127758336,"data_stored":127582870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,3,5,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6240000000000001,"15min":1.6850000000000001},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":7.7460000000000004,"5min":13.266999999999999,"15min":17.408999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":2.2970000000000002,"5min":1.8160000000000001,"15min":2.0339999999999998},"min":{"1min":0.19600000000000001,"5min":0.185,"15min":0.185},"max":{"1min":9.3399999999999999,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.33700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":8.1820000000000004},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.22700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:13:44 2024","interfaces":[{"interface":"back","average":{"1min":5.0910000000000002,"5min":3.6429999999999998,"15min":1.827},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":26.143000000000001,"5min":40.784999999999997,"15min":40.784999999999997},"last":9.7929999999999993},{"interface":"front","average":{"1min":2.0419999999999998,"5min":2.4180000000000001,"15min":1.5089999999999999},"min":{"1min":0.28799999999999998,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":12.036,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.373}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718236,"kb_used_data":124140,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93653604,"statfs":{"total":96636764160,"available":95901290496,"internally_reserved":0,"allocated":127119360,"data_stored":126947040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.29099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":12.867000000000001},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":12.928000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.375},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.35099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433332,"kb_used_data":122976,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93938508,"statfs":{"total":96636764160,"available":96193032192,"internally_reserved":0,"allocated":125927424,"data_stored":125751958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,1,4,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":3.9500000000000002},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.371}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":3.6179999999999999},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":17.347999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.27500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718828,"kb_used_data":124316,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93653012,"statfs":{"total":96636764160,"available":95900684288,"internally_reserved":0,"allocated":127299584,"data_stored":127124118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":5.7549999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.25900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":5.8280000000000003},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.81899999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.432}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":126976000,"data_stored":126976000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":125485056,"data_stored":125485056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":126828544,"data_stored":126828544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":126898176,"data_stored":126898176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:49.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9950", 'reported_seq': 10016, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.122217+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:14:45.122217+0000', 'last_peered': '2024-09-15T03:14:45.122217+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:14:45.122217+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9950, 'log_dups_size': 0, 'ondisk_log_size': 9950, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16056320, 'num_objects': 1960, 'num_object_clones': 0, 'num_object_copies': 7840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9950, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -6}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9588", 'reported_seq': 9682, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.035765+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:14:45.035765+0000', 'last_peered': '2024-09-15T03:14:45.035765+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:14:45.035765+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9588, 'log_dups_size': 0, 'ondisk_log_size': 9588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15054450, 'num_objects': 1838, 'num_object_clones': 0, 'num_object_copies': 7352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1838, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9586, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -146}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10210", 'reported_seq': 10318, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.260681+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:14:45.260681+0000', 'last_peered': '2024-09-15T03:14:45.260681+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:14:45.260681+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16464721, 'num_objects': 2011, 'num_object_clones': 0, 'num_object_copies': 8044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2011, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10209, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -80}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10098", 'reported_seq': 10204, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.021074+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:14:45.021074+0000', 'last_peered': '2024-09-15T03:14:45.021074+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:14:45.021074+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16242290, 'num_objects': 1983, 'num_object_clones': 0, 'num_object_copies': 7932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10096, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -101}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10087", 'reported_seq': 10171, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.208814+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:14:45.208814+0000', 'last_peered': '2024-09-15T03:14:45.208814+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:14:45.208814+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15654912, 'num_objects': 1911, 'num_object_clones': 0, 'num_object_copies': 7644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10087, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -127}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:49.734 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10045", 'reported_seq': 10166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.084585+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:14:45.084585+0000', 'last_peered': '2024-09-15T03:14:45.084585+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:14:45.084585+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15668850, 'num_objects': 1913, 'num_object_clones': 0, 'num_object_copies': 7652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10043, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -113}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10136", 'reported_seq': 10253, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:44.883022+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:14:44.883022+0000', 'last_peered': '2024-09-15T03:14:44.883022+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:14:44.883022+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15963762, 'num_objects': 1949, 'num_object_clones': 0, 'num_object_copies': 7796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10134, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -107}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9990", 'reported_seq': 10081, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:44.841887+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:14:44.841887+0000', 'last_peered': '2024-09-15T03:14:44.841887+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:14:44.841887+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9990, 'log_dups_size': 0, 'ondisk_log_size': 9990, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16215268, 'num_objects': 1980, 'num_object_clones': 0, 'num_object_copies': 7920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9987, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -47}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10014", 'reported_seq': 10133, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.256243+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:14:45.256243+0000', 'last_peered': '2024-09-15T03:14:45.256243+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:14:45.256243+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16004722, 'num_objects': 1954, 'num_object_clones': 0, 'num_object_copies': 7816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10012, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -88}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10016", 'reported_seq': 10110, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.201489+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:14:45.201489+0000', 'last_peered': '2024-09-15T03:14:45.201489+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:14:45.201489+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15290795, 'num_objects': 1867, 'num_object_clones': 0, 'num_object_copies': 7468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1867, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10013, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -195}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:14:49.735 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10120", 'reported_seq': 13906, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.353495+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:14:45.353495+0000', 'last_peered': '2024-09-15T03:14:45.353495+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:14:45.353495+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16489273, 'num_objects': 2013, 'num_object_clones': 0, 'num_object_copies': 8052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 188, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10119, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1853, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:49.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:14:49.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9824", 'reported_seq': 9924, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:44.550373+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:14:44.550373+0000', 'last_peered': '2024-09-15T03:14:44.550373+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:14:44.550373+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9824, 'log_dups_size': 0, 'ondisk_log_size': 9824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15218290, 'num_objects': 1858, 'num_object_clones': 0, 'num_object_copies': 7432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9822, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:14:49.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:14:49.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9978", 'reported_seq': 10085, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.246399+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:14:45.246399+0000', 'last_peered': '2024-09-15T03:14:45.246399+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:14:45.246399+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9978, 'log_dups_size': 0, 'ondisk_log_size': 9978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15603314, 'num_objects': 1905, 'num_object_clones': 0, 'num_object_copies': 7620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9976, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -72}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:14:49.736 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:14:49.736 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10056", 'reported_seq': 10169, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:45.104371+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:14:45.104371+0000', 'last_peered': '2024-09-15T03:14:45.104371+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:14:45.104371+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15996530, 'num_objects': 1953, 'num_object_clones': 0, 'num_object_copies': 7812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10054, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:14:49.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:14:52.252 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:52.253 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26561/120640 objects degraded (22.017%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":615,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30160,"data_bytes":247028308,"bytes_used":2645868544,"bytes_avail":383901188096,"bytes_total":386547056640,"degraded_objects":26561,"degraded_total":120640,"degraded_ratio":0.22016744031830238,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":218},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:14:52.281 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:14:52.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:54.030 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:54.059 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:54.066 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-15T03:14:52.911952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":246200916,"num_objects":30059,"num_object_clones":0,"num_object_copies":120236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160648,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6123,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159875,"ondisk_log_size":159875,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2567952,"kb_used_data":482788,"kb_used_omap":14,"kb_used_meta":2085105,"kb_avail":374919408,"statfs":{"total":386547056640,"available":383917473792,"internally_reserved":0,"allocated":494374912,"data_stored":493676194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135147547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,4,10,13,13,7,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15917056,"num_objects":-1943,"num_object_clones":0,"num_object_copies":-7772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1943,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009503"},"pg_stats":[{"pgid":"1.f","version":"35'10013","reported_seq":10079,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.245973+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:50.245973+0000","last_peered":"2024-09-15T03:14:50.245973+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:50.245973+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15540224,"num_objects":1897,"num_object_clones":0,"num_object_copies":7588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10013,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-69}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9648","reported_seq":9742,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.111815+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:50.111815+0000","last_peered":"2024-09-15T03:14:50.111815+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:50.111815+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9648,"log_dups_size":0,"ondisk_log_size":9648,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14562930,"num_objects":1778,"num_object_clones":0,"num_object_copies":7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9646,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-206}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10269","reported_seq":10377,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.280219+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:50.280219+0000","last_peered":"2024-09-15T03:14:50.280219+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:50.280219+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15981393,"num_objects":1952,"num_object_clones":0,"num_object_copies":7808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10268,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-139}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9757","reported_seq":13729,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:52.587364+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:52.587364+0000","last_peered":"2024-09-15T03:14:52.587364+0000","last_clean":"2024-09-15T03:14:52.587364+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:52.587364+0000","last_undegraded":"2024-09-15T03:14:52.587364+0000","last_fullsized":"2024-09-15T03:14:52.587364+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9757,"log_dups_size":0,"ondisk_log_size":9757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15324786,"num_objects":1871,"num_object_clones":0,"num_object_copies":7484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'10171","reported_seq":10276,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.063254+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:50.063254+0000","last_peered":"2024-09-15T03:14:50.063254+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:50.063254+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15644274,"num_objects":1910,"num_object_clones":0,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10169,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-174}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10136","reported_seq":10220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.230433+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:50.230433+0000","last_peered":"2024-09-15T03:14:50.230433+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:50.230433+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15253504,"num_objects":1862,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10136,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10102","reported_seq":10223,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.200208+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:50.200208+0000","last_peered":"2024-09-15T03:14:50.200208+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:50.200208+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15201906,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10100,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-170}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10190","reported_seq":10307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.024310+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:50.024310+0000","last_peered":"2024-09-15T03:14:50.024310+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:50.024310+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15521394,"num_objects":1895,"num_object_clones":0,"num_object_copies":7580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10188,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-161}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10047","reported_seq":10138,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.166723+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:50.166723+0000","last_peered":"2024-09-15T03:14:50.166723+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:50.166723+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15748324,"num_objects":1923,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-104}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10071","reported_seq":10190,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.294363+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:50.294363+0000","last_peered":"2024-09-15T03:14:50.294363+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:50.294363+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15537778,"num_objects":1897,"num_object_clones":0,"num_object_copies":7588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-145}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10067","reported_seq":10159,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.145028+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:50.145028+0000","last_peered":"2024-09-15T03:14:50.145028+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:50.145028+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14873003,"num_objects":1816,"num_object_clones":0,"num_object_copies":7264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10064,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10017","reported_seq":14058,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:52.591567+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:52.591567+0000","last_peered":"2024-09-15T03:14:52.591567+0000","last_clean":"2024-09-15T03:14:52.591567+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:52.591567+0000","last_undegraded":"2024-09-15T03:14:52.591567+0000","last_fullsized":"2024-09-15T03:14:52.591567+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15416121,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10175","reported_seq":14111,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.492161+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:50.492161+0000","last_peered":"2024-09-15T03:14:50.492161+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:50.492161+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16038713,"num_objects":1958,"num_object_clones":0,"num_object_copies":7832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1845}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9873","reported_seq":9973,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.231520+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:50.231520+0000","last_peered":"2024-09-15T03:14:50.231520+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:50.231520+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9873,"log_dups_size":0,"ondisk_log_size":9873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816882,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9871,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-204}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10027","reported_seq":10134,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.105823+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:50.105823+0000","last_peered":"2024-09-15T03:14:50.105823+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:50.105823+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15201906,"num_objects":1856,"num_object_clones":0,"num_object_copies":7424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-121}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10112","reported_seq":10225,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:50.187364+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:50.187364+0000","last_peered":"2024-09-15T03:14:50.187364+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:50.187364+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15537778,"num_objects":1897,"num_object_clones":0,"num_object_copies":7588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10110,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":246200916,"num_objects":30059,"num_object_clones":0,"num_object_copies":120236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160648,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6123,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491446272,"data_stored":491446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159875,"ondisk_log_size":159875,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711596,"kb_used_data":121188,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93660244,"statfs":{"total":96636764160,"available":95908089856,"internally_reserved":0,"allocated":124096512,"data_stored":123921046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,5,5,1,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.35399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":4.8490000000000002},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":4.7569999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.73199999999999998},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.28899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718824,"kb_used_data":120608,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93653016,"statfs":{"total":96636764160,"available":95900688384,"internally_reserved":0,"allocated":123502592,"data_stored":123330272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,3,1,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":19.388000000000002},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":19.189}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.437},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.46700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.42299999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417428,"kb_used_data":119552,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93954412,"statfs":{"total":96636764160,"available":96209317888,"internally_reserved":0,"allocated":122421248,"data_stored":122245782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":7.2670000000000003},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.36699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":7.3120000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.33600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720104,"kb_used_data":121440,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93651736,"statfs":{"total":96636764160,"available":95899377664,"internally_reserved":0,"allocated":124354560,"data_stored":124179094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,2,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":6.202},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.2830000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.23100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:13:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8200000000000001,"5min":1.6220000000000001,"15min":1.2490000000000001},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":14.339,"5min":20.199000000000002,"15min":20.948},"last":0.28199999999999997},{"interface":"front","average":{"1min":2.2549999999999999,"5min":2.2999999999999998,"15min":1.25},"min":{"1min":0.26700000000000002,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":14.422000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.67900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":123330560,"data_stored":123330560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121745408,"data_stored":121745408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":123113472,"data_stored":123113472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":123256832,"data_stored":123256832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:57.069 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:14:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:14:58.967 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:14:59.006 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-09-15T03:14:58.913358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":238647892,"num_objects":29137,"num_object_clones":0,"num_object_copies":116548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161570,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6182,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160097,"ondisk_log_size":160097,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2544576,"kb_used_data":467744,"kb_used_omap":14,"kb_used_meta":2076785,"kb_avail":374942784,"statfs":{"total":386547056640,"available":383941410816,"internally_reserved":0,"allocated":478969856,"data_stored":478271138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126627867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,3,17,16,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15826944,"num_objects":-1932,"num_object_clones":0,"num_object_copies":-7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1932,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005965"},"pg_stats":[{"pgid":"1.f","version":"35'10082","reported_seq":10148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.043163+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:55.043163+0000","last_peered":"2024-09-15T03:14:55.043163+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:55.043163+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14974976,"num_objects":1828,"num_object_clones":0,"num_object_copies":7312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9700","reported_seq":9794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:54.985606+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:54.985606+0000","last_peered":"2024-09-15T03:14:54.985606+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:54.985606+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9700,"log_dups_size":0,"ondisk_log_size":9700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14136946,"num_objects":1726,"num_object_clones":0,"num_object_copies":6904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9698,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10331","reported_seq":10439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.365854+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:55.365854+0000","last_peered":"2024-09-15T03:14:55.365854+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:55.365854+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15473489,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10330,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-201}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9819","reported_seq":13791,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:57.620018+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:57.620018+0000","last_peered":"2024-09-15T03:14:57.620018+0000","last_clean":"2024-09-15T03:14:57.620018+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:57.620018+0000","last_undegraded":"2024-09-15T03:14:57.620018+0000","last_fullsized":"2024-09-15T03:14:57.620018+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816882,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'10237","reported_seq":10343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.189186+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:55.189186+0000","last_peered":"2024-09-15T03:14:55.189186+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:55.189186+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15103602,"num_objects":1844,"num_object_clones":0,"num_object_copies":7376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10235,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10195","reported_seq":10278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.360676+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:55.360676+0000","last_peered":"2024-09-15T03:14:55.360676+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:55.360676+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14770176,"num_objects":1803,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10195,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10151","reported_seq":10272,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:54.881137+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:54.881137+0000","last_peered":"2024-09-15T03:14:54.881137+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:54.881137+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14800498,"num_objects":1807,"num_object_clones":0,"num_object_copies":7228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10149,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-219}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10240","reported_seq":10357,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.067063+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:55.067063+0000","last_peered":"2024-09-15T03:14:55.067063+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:55.067063+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15111794,"num_objects":1845,"num_object_clones":0,"num_object_copies":7380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10238,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10109","reported_seq":10200,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.214856+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:55.214856+0000","last_peered":"2024-09-15T03:14:55.214856+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:55.214856+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15240420,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10131","reported_seq":10249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.543149+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:55.543149+0000","last_peered":"2024-09-15T03:14:55.543149+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:55.543149+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15046258,"num_objects":1837,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10129,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10118","reported_seq":10212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.344497+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:55.344497+0000","last_peered":"2024-09-15T03:14:55.344497+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:55.344497+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14455211,"num_objects":1765,"num_object_clones":0,"num_object_copies":7060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10115,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-297}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10083","reported_seq":14124,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:57.708577+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:57.708577+0000","last_peered":"2024-09-15T03:14:57.708577+0000","last_clean":"2024-09-15T03:14:57.708577+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:57.708577+0000","last_undegraded":"2024-09-15T03:14:57.708577+0000","last_fullsized":"2024-09-15T03:14:57.708577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14875449,"num_objects":1816,"num_object_clones":0,"num_object_copies":7264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10229","reported_seq":14283,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.503249+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:55.503249+0000","last_peered":"2024-09-15T03:14:55.503249+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:55.503249+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15596345,"num_objects":1904,"num_object_clones":0,"num_object_copies":7616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1987,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1850}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9921","reported_seq":10021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.106281+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:55.106281+0000","last_peered":"2024-09-15T03:14:55.106281+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:55.106281+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14423666,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10074","reported_seq":10180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.452683+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:55.452683+0000","last_peered":"2024-09-15T03:14:55.452683+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:55.452683+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816882,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10072,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10177","reported_seq":10290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.075152+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:55.075152+0000","last_peered":"2024-09-15T03:14:55.075152+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:55.075152+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15005298,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10175,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":238647892,"num_objects":29137,"num_object_clones":0,"num_object_copies":116548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161570,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6182,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476725248,"data_stored":476725248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160097,"ondisk_log_size":160097,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716388,"kb_used_data":117660,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93655452,"statfs":{"total":96636764160,"available":95903182848,"internally_reserved":0,"allocated":120483840,"data_stored":120308374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,6,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":5.4569999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":5.4900000000000002},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.25800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":11.444000000000001},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.28100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702728,"kb_used_data":116992,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93669112,"statfs":{"total":96636764160,"available":95917170688,"internally_reserved":0,"allocated":119799808,"data_stored":119627488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":8.0800000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":10.257},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":10.289}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.85499999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417916,"kb_used_data":115892,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93953924,"statfs":{"total":96636764160,"available":96208818176,"internally_reserved":0,"allocated":118673408,"data_stored":118497942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":6.5999999999999996},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.29599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.38200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707544,"kb_used_data":117200,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93664296,"statfs":{"total":96636764160,"available":95912239104,"internally_reserved":0,"allocated":120012800,"data_stored":119837334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.78400000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.74199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119697408,"data_stored":119697408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118005760,"data_stored":118005760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":119447552,"data_stored":119447552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":119574528,"data_stored":119574528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:14:59.007 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:00.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:00.738 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:00.765 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-15T03:15:00.096510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":238647892,"num_objects":29137,"num_object_clones":0,"num_object_copies":116548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161570,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6182,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160097,"ondisk_log_size":160097,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2544576,"kb_used_data":467744,"kb_used_omap":14,"kb_used_meta":2076785,"kb_avail":374942784,"statfs":{"total":386547056640,"available":383941410816,"internally_reserved":0,"allocated":478969856,"data_stored":478271138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126627867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4,3,17,16,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14843904,"num_objects":-1812,"num_object_clones":0,"num_object_copies":-7248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1812,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.188559"},"pg_stats":[{"pgid":"1.f","version":"35'10082","reported_seq":10148,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.043163+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:14:55.043163+0000","last_peered":"2024-09-15T03:14:55.043163+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:14:55.043163+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14974976,"num_objects":1828,"num_object_clones":0,"num_object_copies":7312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9700","reported_seq":9794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:54.985606+0000","last_change":"2024-09-15T03:06:09.205627+0000","last_active":"2024-09-15T03:14:54.985606+0000","last_peered":"2024-09-15T03:14:54.985606+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:14:54.985606+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9700,"log_dups_size":0,"ondisk_log_size":9700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14136946,"num_objects":1726,"num_object_clones":0,"num_object_copies":6904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9698,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'10331","reported_seq":10439,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.365854+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:14:55.365854+0000","last_peered":"2024-09-15T03:14:55.365854+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:14:55.365854+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15473489,"num_objects":1890,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10330,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-201}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9819","reported_seq":13791,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:57.620018+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:14:57.620018+0000","last_peered":"2024-09-15T03:14:57.620018+0000","last_clean":"2024-09-15T03:14:57.620018+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:14:57.620018+0000","last_undegraded":"2024-09-15T03:14:57.620018+0000","last_fullsized":"2024-09-15T03:14:57.620018+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9819,"log_dups_size":0,"ondisk_log_size":9819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816882,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9817,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"35'10237","reported_seq":10343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.189186+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:14:55.189186+0000","last_peered":"2024-09-15T03:14:55.189186+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:14:55.189186+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15103602,"num_objects":1844,"num_object_clones":0,"num_object_copies":7376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10235,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10195","reported_seq":10278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.360676+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:14:55.360676+0000","last_peered":"2024-09-15T03:14:55.360676+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:14:55.360676+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14770176,"num_objects":1803,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10195,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-235}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10151","reported_seq":10272,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:54.881137+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:14:54.881137+0000","last_peered":"2024-09-15T03:14:54.881137+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:14:54.881137+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14800498,"num_objects":1807,"num_object_clones":0,"num_object_copies":7228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10149,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-219}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10240","reported_seq":10357,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.067063+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:14:55.067063+0000","last_peered":"2024-09-15T03:14:55.067063+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:14:55.067063+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15111794,"num_objects":1845,"num_object_clones":0,"num_object_copies":7380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10238,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10109","reported_seq":10200,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.214856+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:14:55.214856+0000","last_peered":"2024-09-15T03:14:55.214856+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:14:55.214856+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15240420,"num_objects":1861,"num_object_clones":0,"num_object_copies":7444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10106,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10131","reported_seq":10249,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.543149+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:14:55.543149+0000","last_peered":"2024-09-15T03:14:55.543149+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:14:55.543149+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15046258,"num_objects":1837,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10129,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10118","reported_seq":10212,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.344497+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:14:55.344497+0000","last_peered":"2024-09-15T03:14:55.344497+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:14:55.344497+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14455211,"num_objects":1765,"num_object_clones":0,"num_object_copies":7060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10115,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-297}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10083","reported_seq":14124,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T03:14:57.708577+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:14:57.708577+0000","last_peered":"2024-09-15T03:14:57.708577+0000","last_clean":"2024-09-15T03:14:57.708577+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:14:57.708577+0000","last_undegraded":"2024-09-15T03:14:57.708577+0000","last_fullsized":"2024-09-15T03:14:57.708577+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14875449,"num_objects":1816,"num_object_clones":0,"num_object_copies":7264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10082,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"35'10229","reported_seq":14283,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.503249+0000","last_change":"2024-09-15T03:11:58.381322+0000","last_active":"2024-09-15T03:14:55.503249+0000","last_peered":"2024-09-15T03:14:55.503249+0000","last_clean":"2024-09-15T03:04:56.842706+0000","last_became_active":"2024-09-15T03:06:09.175449+0000","last_became_peered":"2024-09-15T03:06:09.175449+0000","last_unstale":"2024-09-15T03:14:55.503249+0000","last_undegraded":"2024-09-15T03:06:08.892453+0000","last_fullsized":"2024-09-15T03:06:08.836310+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15596345,"num_objects":1904,"num_object_clones":0,"num_object_copies":7616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1987,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1850}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9921","reported_seq":10021,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.106281+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:14:55.106281+0000","last_peered":"2024-09-15T03:14:55.106281+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:14:55.106281+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14423666,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-252}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10074","reported_seq":10180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.452683+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:14:55.452683+0000","last_peered":"2024-09-15T03:14:55.452683+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:14:55.452683+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14816882,"num_objects":1809,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10072,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10177","reported_seq":10290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:14:55.075152+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:14:55.075152+0000","last_peered":"2024-09-15T03:14:55.075152+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:14:55.075152+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15005298,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10175,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-227}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":238647892,"num_objects":29137,"num_object_clones":0,"num_object_copies":116548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29137,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161570,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6182,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476725248,"data_stored":476725248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160097,"ondisk_log_size":160097,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716388,"kb_used_data":117660,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93655452,"statfs":{"total":96636764160,"available":95903182848,"internally_reserved":0,"allocated":120483840,"data_stored":120308374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,9,6,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.22900000000000001},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":5.4569999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":5.4900000000000002},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.25800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":11.444000000000001},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.28100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702728,"kb_used_data":116992,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93669112,"statfs":{"total":96636764160,"available":95917170688,"internally_reserved":0,"allocated":119799808,"data_stored":119627488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":8.0800000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":10.257},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":10.289}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.85499999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417916,"kb_used_data":115892,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93953924,"statfs":{"total":96636764160,"available":96208818176,"internally_reserved":0,"allocated":118673408,"data_stored":118497942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":6.5999999999999996},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.29599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.23200000000000001},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":0.39900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.38200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707544,"kb_used_data":117200,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93664296,"statfs":{"total":96636764160,"available":95912239104,"internally_reserved":0,"allocated":120012800,"data_stored":119837334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.78400000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.74199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119697408,"data_stored":119697408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118005760,"data_stored":118005760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":119447552,"data_stored":119447552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":119574528,"data_stored":119574528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:15:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10082", 'reported_seq': 10148, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.043163+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:14:55.043163+0000', 'last_peered': '2024-09-15T03:14:55.043163+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:14:55.043163+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14974976, 'num_objects': 1828, 'num_object_clones': 0, 'num_object_copies': 7312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1828, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10082, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -138}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:00.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9700", 'reported_seq': 9794, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:54.985606+0000', 'last_change': '2024-09-15T03:06:09.205627+0000', 'last_active': '2024-09-15T03:14:54.985606+0000', 'last_peered': '2024-09-15T03:14:54.985606+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:14:54.985606+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9700, 'log_dups_size': 0, 'ondisk_log_size': 9700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14136946, 'num_objects': 1726, 'num_object_clones': 0, 'num_object_copies': 6904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9698, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -258}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10331", 'reported_seq': 10439, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.365854+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:14:55.365854+0000', 'last_peered': '2024-09-15T03:14:55.365854+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:14:55.365854+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15473489, 'num_objects': 1890, 'num_object_clones': 0, 'num_object_copies': 7560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1890, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10330, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -201}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10237", 'reported_seq': 10343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.189186+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:14:55.189186+0000', 'last_peered': '2024-09-15T03:14:55.189186+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:14:55.189186+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15103602, 'num_objects': 1844, 'num_object_clones': 0, 'num_object_copies': 7376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10235, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -240}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10195", 'reported_seq': 10278, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.360676+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:14:55.360676+0000', 'last_peered': '2024-09-15T03:14:55.360676+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:14:55.360676+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14770176, 'num_objects': 1803, 'num_object_clones': 0, 'num_object_copies': 7212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10195, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -235}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10151", 'reported_seq': 10272, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:54.881137+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:14:54.881137+0000', 'last_peered': '2024-09-15T03:14:54.881137+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:14:54.881137+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14800498, 'num_objects': 1807, 'num_object_clones': 0, 'num_object_copies': 7228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10149, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -219}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:15:00.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10240", 'reported_seq': 10357, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.067063+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:14:55.067063+0000', 'last_peered': '2024-09-15T03:14:55.067063+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:14:55.067063+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15111794, 'num_objects': 1845, 'num_object_clones': 0, 'num_object_copies': 7380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10238, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -211}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10109", 'reported_seq': 10200, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.214856+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:14:55.214856+0000', 'last_peered': '2024-09-15T03:14:55.214856+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:14:55.214856+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15240420, 'num_objects': 1861, 'num_object_clones': 0, 'num_object_copies': 7444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10106, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -166}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10131", 'reported_seq': 10249, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.543149+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:14:55.543149+0000', 'last_peered': '2024-09-15T03:14:55.543149+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:14:55.543149+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15046258, 'num_objects': 1837, 'num_object_clones': 0, 'num_object_copies': 7348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10129, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -205}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "35'10118", 'reported_seq': 10212, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.344497+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:14:55.344497+0000', 'last_peered': '2024-09-15T03:14:55.344497+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:14:55.344497+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14455211, 'num_objects': 1765, 'num_object_clones': 0, 'num_object_copies': 7060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10115, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -297}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10229", 'reported_seq': 14283, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.503249+0000', 'last_change': '2024-09-15T03:11:58.381322+0000', 'last_active': '2024-09-15T03:14:55.503249+0000', 'last_peered': '2024-09-15T03:14:55.503249+0000', 'last_clean': '2024-09-15T03:04:56.842706+0000', 'last_became_active': '2024-09-15T03:06:09.175449+0000', 'last_became_peered': '2024-09-15T03:06:09.175449+0000', 'last_unstale': '2024-09-15T03:14:55.503249+0000', 'last_undegraded': '2024-09-15T03:06:08.892453+0000', 'last_fullsized': '2024-09-15T03:06:08.836310+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15596345, 'num_objects': 1904, 'num_object_clones': 0, 'num_object_copies': 7616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 54, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10228, 'num_write_kb': 48527, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1987, 'num_bytes_recovered': 163840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1850}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:15:00.768 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9921", 'reported_seq': 10021, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.106281+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:14:55.106281+0000', 'last_peered': '2024-09-15T03:14:55.106281+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:14:55.106281+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9921, 'log_dups_size': 0, 'ondisk_log_size': 9921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14423666, 'num_objects': 1761, 'num_object_clones': 0, 'num_object_copies': 7044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9919, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:00.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:15:00.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10074", 'reported_seq': 10180, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.452683+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:14:55.452683+0000', 'last_peered': '2024-09-15T03:14:55.452683+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:14:55.452683+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14816882, 'num_objects': 1809, 'num_object_clones': 0, 'num_object_copies': 7236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10072, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:00.769 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:15:00.769 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10177", 'reported_seq': 10290, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:14:55.075152+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:14:55.075152+0000', 'last_peered': '2024-09-15T03:14:55.075152+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:14:55.075152+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15005298, 'num_objects': 1832, 'num_object_clones': 0, 'num_object_copies': 7328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10175, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -227}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:00.769 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:15:02.459 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:02.459 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26499/113688 objects degraded (23.309%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":625,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28422,"data_bytes":232790612,"bytes_used":2608189440,"bytes_avail":383938867200,"bytes_total":386547056640,"degraded_objects":26499,"degraded_total":113688,"degraded_ratio":0.23308528604602072,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":173},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:15:02.492 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:15:02.493 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:03.913 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:03.927 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:03.953 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-15T03:15:02.914323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231160404,"num_objects":28223,"num_object_clones":0,"num_object_copies":112892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162484,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6239,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160311,"ondisk_log_size":160311,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2547708,"kb_used_data":454208,"kb_used_omap":14,"kb_used_meta":2093425,"kb_avail":374939652,"statfs":{"total":386547056640,"available":383938203648,"internally_reserved":0,"allocated":465108992,"data_stored":464397752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143667227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,1,4,11,17,18,18,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15040512,"num_objects":-1836,"num_object_clones":0,"num_object_copies":-7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1836,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001918"},"pg_stats":[{"pgid":"1.f","version":"36'10133","reported_seq":10200,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.234798+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:00.234798+0000","last_peered":"2024-09-15T03:15:00.234798+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:00.234798+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14557184,"num_objects":1777,"num_object_clones":0,"num_object_copies":7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10133,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-189}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"36'9749","reported_seq":9851,"reported_epoch":36,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.231073+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:00.231073+0000","last_peered":"2024-09-15T03:15:00.231073+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:00.231073+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9749,"log_dups_size":0,"ondisk_log_size":9749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13735538,"num_objects":1677,"num_object_clones":0,"num_object_copies":6708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9747,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-306}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"36'10396","reported_seq":10504,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.441135+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:00.441135+0000","last_peered":"2024-09-15T03:15:00.441135+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:00.441135+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14941009,"num_objects":1825,"num_object_clones":0,"num_object_copies":7300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10395,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-266}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9879","reported_seq":13853,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:02.827284+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:02.827284+0000","last_peered":"2024-09-15T03:15:02.827284+0000","last_clean":"2024-09-15T03:15:02.827284+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:02.827284+0000","last_undegraded":"2024-09-15T03:15:02.827284+0000","last_fullsized":"2024-09-15T03:15:02.827284+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9879,"log_dups_size":0,"ondisk_log_size":9879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14325362,"num_objects":1749,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9877,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"36'10287","reported_seq":10394,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.097927+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:00.097927+0000","last_peered":"2024-09-15T03:15:00.097927+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:00.097927+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14694002,"num_objects":1794,"num_object_clones":0,"num_object_copies":7176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10285,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-290}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"36'10247","reported_seq":10330,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.122227+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:00.122227+0000","last_peered":"2024-09-15T03:15:00.122227+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:00.122227+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14344192,"num_objects":1751,"num_object_clones":0,"num_object_copies":7004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10247,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10206","reported_seq":10328,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.102917+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:00.102917+0000","last_peered":"2024-09-15T03:15:00.102917+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:00.102917+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14349938,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-274}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"36'10298","reported_seq":10416,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.103294+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:00.103294+0000","last_peered":"2024-09-15T03:15:00.103294+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:00.103294+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14636658,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10296,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"36'10183","reported_seq":10274,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.167513+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:00.167513+0000","last_peered":"2024-09-15T03:15:00.167513+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:00.167513+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14634212,"num_objects":1787,"num_object_clones":0,"num_object_copies":7148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10180,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"36'10180","reported_seq":10300,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.296102+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:00.296102+0000","last_peered":"2024-09-15T03:15:00.296102+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:00.296102+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14644850,"num_objects":1788,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10178,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-254}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"36'10166","reported_seq":10261,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.222302+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:00.222302+0000","last_peered":"2024-09-15T03:15:00.222302+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:00.222302+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14061995,"num_objects":1717,"num_object_clones":0,"num_object_copies":6868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10163,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10138","reported_seq":14181,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:02.601713+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:02.601713+0000","last_peered":"2024-09-15T03:15:02.601713+0000","last_clean":"2024-09-15T03:15:02.601713+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:02.601713+0000","last_undegraded":"2024-09-15T03:15:02.601713+0000","last_fullsized":"2024-09-15T03:15:02.601713+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14424889,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10137,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10313","reported_seq":14485,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:02.827751+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:02.827751+0000","last_peered":"2024-09-15T03:15:02.827751+0000","last_clean":"2024-09-15T03:15:02.827751+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:02.827751+0000","last_undegraded":"2024-09-15T03:15:02.827751+0000","last_fullsized":"2024-09-15T03:15:02.827751+0000","mapping_epoch":36,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14908217,"num_objects":1820,"num_object_clones":0,"num_object_copies":7280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10312,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"36'9972","reported_seq":10073,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.207377+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:00.207377+0000","last_peered":"2024-09-15T03:15:00.207377+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:00.207377+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9972,"log_dups_size":0,"ondisk_log_size":9972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14005874,"num_objects":1710,"num_object_clones":0,"num_object_copies":6840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9970,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"36'10134","reported_seq":10242,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.493335+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:00.493335+0000","last_peered":"2024-09-15T03:15:00.493335+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:00.493335+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14325362,"num_objects":1749,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10132,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-228}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10230","reported_seq":10344,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:00.103308+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:00.103308+0000","last_peered":"2024-09-15T03:15:00.103308+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:00.103308+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14571122,"num_objects":1779,"num_object_clones":0,"num_object_copies":7116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":231160404,"num_objects":28223,"num_object_clones":0,"num_object_copies":112892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162484,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6239,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":461975552,"data_stored":461975552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160311,"ondisk_log_size":160311,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716984,"kb_used_data":114096,"kb_used_omap":4,"kb_used_meta":602875,"kb_avail":93654856,"statfs":{"total":96636764160,"available":95902572544,"internally_reserved":0,"allocated":116834304,"data_stored":116658838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":617344577},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,5,9,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.41399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":12.286},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":4.5510000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.52900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703364,"kb_used_data":113464,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93668476,"statfs":{"total":96636764160,"available":95916519424,"internally_reserved":0,"allocated":116187136,"data_stored":116010649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,8,3,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.441},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.47099999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.49299999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.51400000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":418564,"kb_used_data":112380,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93953276,"statfs":{"total":96636764160,"available":96208154624,"internally_reserved":0,"allocated":115077120,"data_stored":114893299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,0,6,5,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":5.2450000000000001},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":6.6280000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.379},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":5.2759999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":6.5949999999999998},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":0.40699999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708796,"kb_used_data":114268,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93663044,"statfs":{"total":96636764160,"available":95910957056,"internally_reserved":0,"allocated":117010432,"data_stored":116834966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,3,3,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.496}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.26500000000000001},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115974144,"data_stored":115974144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114434048,"data_stored":114434048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115671040,"data_stored":115671040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115896320,"data_stored":115896320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:06.955 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:08.968 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:08.981 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:08.985 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-15T03:15:08.917375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":223967828,"num_objects":27345,"num_object_clones":0,"num_object_copies":109380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163362,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6279,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160489,"ondisk_log_size":160489,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2533092,"kb_used_data":439620,"kb_used_omap":14,"kb_used_meta":2093425,"kb_avail":374954268,"statfs":{"total":386547056640,"available":383953170432,"internally_reserved":0,"allocated":450170880,"data_stored":449438742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143667227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,6,9,9,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14680064,"num_objects":-1792,"num_object_clones":0,"num_object_copies":-7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1792,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.820865"},"pg_stats":[{"pgid":"1.f","version":"37'10195","reported_seq":10262,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.435307+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:05.435307+0000","last_peered":"2024-09-15T03:15:05.435307+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:05.435307+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14049280,"num_objects":1715,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10195,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9818","reported_seq":10001,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.235108+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:05.235108+0000","last_peered":"2024-09-15T03:15:05.235108+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:05.235108+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13170290,"num_objects":1608,"num_object_clones":0,"num_object_copies":6432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10441","reported_seq":10551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.324466+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:05.324466+0000","last_peered":"2024-09-15T03:15:05.324466+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:05.324466+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14572369,"num_objects":1780,"num_object_clones":0,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10440,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-311}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9958","reported_seq":13932,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:08.156427+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:08.156427+0000","last_peered":"2024-09-15T03:15:08.156427+0000","last_clean":"2024-09-15T03:15:08.156427+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:08.156427+0000","last_undegraded":"2024-09-15T03:15:08.156427+0000","last_fullsized":"2024-09-15T03:15:08.156427+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9958,"log_dups_size":0,"ondisk_log_size":9958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13678194,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9956,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10341","reported_seq":10449,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.406475+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:05.406475+0000","last_peered":"2024-09-15T03:15:05.406475+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:05.406475+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14251634,"num_objects":1740,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10299","reported_seq":10385,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.302777+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:05.302777+0000","last_peered":"2024-09-15T03:15:05.302777+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:05.302777+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13918208,"num_objects":1699,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10299,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10252","reported_seq":10375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.009044+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:05.009044+0000","last_peered":"2024-09-15T03:15:05.009044+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:05.009044+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13973106,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10250,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-320}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10343","reported_seq":10462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.179640+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:05.179640+0000","last_peered":"2024-09-15T03:15:05.179640+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:05.179640+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14268018,"num_objects":1742,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10229","reported_seq":10322,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.405109+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:05.405109+0000","last_peered":"2024-09-15T03:15:05.405109+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:05.405109+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14257380,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10231","reported_seq":10352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.673726+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:05.673726+0000","last_peered":"2024-09-15T03:15:05.673726+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:05.673726+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14227058,"num_objects":1737,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10229,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10215","reported_seq":10311,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:04.962588+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:04.962588+0000","last_peered":"2024-09-15T03:15:04.962588+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:04.962588+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13660587,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10203","reported_seq":14246,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:07.923082+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:07.923082+0000","last_peered":"2024-09-15T03:15:07.923082+0000","last_clean":"2024-09-15T03:15:07.923082+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:07.923082+0000","last_undegraded":"2024-09-15T03:15:07.923082+0000","last_fullsized":"2024-09-15T03:15:07.923082+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13892409,"num_objects":1696,"num_object_clones":0,"num_object_copies":6784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10202,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10381","reported_seq":14557,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:08.171711+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:08.171711+0000","last_peered":"2024-09-15T03:15:08.171711+0000","last_clean":"2024-09-15T03:15:08.171711+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:08.171711+0000","last_undegraded":"2024-09-15T03:15:08.171711+0000","last_fullsized":"2024-09-15T03:15:08.171711+0000","mapping_epoch":36,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14351161,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10380,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10019","reported_seq":10121,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.329761+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:05.329761+0000","last_peered":"2024-09-15T03:15:05.329761+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:05.329761+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13620850,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10186","reported_seq":10295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.609547+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:05.609547+0000","last_peered":"2024-09-15T03:15:05.609547+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:05.609547+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13899378,"num_objects":1697,"num_object_clones":0,"num_object_copies":6788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10184,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-280}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10278","reported_seq":10393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:04.914641+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:04.914641+0000","last_peered":"2024-09-15T03:15:04.914641+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:04.914641+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14177906,"num_objects":1731,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":223967828,"num_objects":27345,"num_object_clones":0,"num_object_copies":109380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163362,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6279,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":447442944,"data_stored":447442944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160489,"ondisk_log_size":160489,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700408,"kb_used_data":110000,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93671432,"statfs":{"total":96636764160,"available":95919546368,"internally_reserved":0,"allocated":112640000,"data_stored":112456179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.8520000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.30399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":110044,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":112685056,"data_stored":112504381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.87},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.48899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.70099999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419188,"kb_used_data":108844,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93952652,"statfs":{"total":96636764160,"available":96207515648,"internally_reserved":0,"allocated":111456256,"data_stored":111272435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":8.3369999999999997},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":9.0519999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":9.0229999999999997},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":9.1639999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":8.9969999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":8.9789999999999992}]}]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709396,"kb_used_data":110732,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93662444,"statfs":{"total":96636764160,"available":95910342656,"internally_reserved":0,"allocated":113389568,"data_stored":113205747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.19900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.28899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":112459776,"data_stored":112459776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110575616,"data_stored":110575616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":112119808,"data_stored":112119808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112287744,"data_stored":112287744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:08.987 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:10.892 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:10.972 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:10.994 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-15T03:15:08.917375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":223967828,"num_objects":27345,"num_object_clones":0,"num_object_copies":109380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163362,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6279,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160489,"ondisk_log_size":160489,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2533092,"kb_used_data":439620,"kb_used_omap":14,"kb_used_meta":2093425,"kb_avail":374954268,"statfs":{"total":386547056640,"available":383953170432,"internally_reserved":0,"allocated":450170880,"data_stored":449438742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143667227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,6,9,9,10,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14680064,"num_objects":-1792,"num_object_clones":0,"num_object_copies":-7168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1792,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.820865"},"pg_stats":[{"pgid":"1.f","version":"37'10195","reported_seq":10262,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.435307+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:05.435307+0000","last_peered":"2024-09-15T03:15:05.435307+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:05.435307+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14049280,"num_objects":1715,"num_object_clones":0,"num_object_copies":6860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10195,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-251}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9818","reported_seq":10001,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.235108+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:05.235108+0000","last_peered":"2024-09-15T03:15:05.235108+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:05.235108+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9818,"log_dups_size":0,"ondisk_log_size":9818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13170290,"num_objects":1608,"num_object_clones":0,"num_object_copies":6432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10441","reported_seq":10551,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.324466+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:05.324466+0000","last_peered":"2024-09-15T03:15:05.324466+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:05.324466+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14572369,"num_objects":1780,"num_object_clones":0,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10440,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-311}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9958","reported_seq":13932,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:08.156427+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:08.156427+0000","last_peered":"2024-09-15T03:15:08.156427+0000","last_clean":"2024-09-15T03:15:08.156427+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:08.156427+0000","last_undegraded":"2024-09-15T03:15:08.156427+0000","last_fullsized":"2024-09-15T03:15:08.156427+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9958,"log_dups_size":0,"ondisk_log_size":9958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13678194,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9956,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10341","reported_seq":10449,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.406475+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:05.406475+0000","last_peered":"2024-09-15T03:15:05.406475+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:05.406475+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14251634,"num_objects":1740,"num_object_clones":0,"num_object_copies":6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10299","reported_seq":10385,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.302777+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:05.302777+0000","last_peered":"2024-09-15T03:15:05.302777+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:05.302777+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13918208,"num_objects":1699,"num_object_clones":0,"num_object_copies":6796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10299,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10252","reported_seq":10375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.009044+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:05.009044+0000","last_peered":"2024-09-15T03:15:05.009044+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:05.009044+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13973106,"num_objects":1706,"num_object_clones":0,"num_object_copies":6824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10250,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-320}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10343","reported_seq":10462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.179640+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:05.179640+0000","last_peered":"2024-09-15T03:15:05.179640+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:05.179640+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14268018,"num_objects":1742,"num_object_clones":0,"num_object_copies":6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-314}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10229","reported_seq":10322,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.405109+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:05.405109+0000","last_peered":"2024-09-15T03:15:05.405109+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:05.405109+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14257380,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10231","reported_seq":10352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.673726+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:05.673726+0000","last_peered":"2024-09-15T03:15:05.673726+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:05.673726+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14227058,"num_objects":1737,"num_object_clones":0,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10229,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10215","reported_seq":10311,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:04.962588+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:04.962588+0000","last_peered":"2024-09-15T03:15:04.962588+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:04.962588+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13660587,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10212,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10203","reported_seq":14246,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:07.923082+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:07.923082+0000","last_peered":"2024-09-15T03:15:07.923082+0000","last_clean":"2024-09-15T03:15:07.923082+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:07.923082+0000","last_undegraded":"2024-09-15T03:15:07.923082+0000","last_fullsized":"2024-09-15T03:15:07.923082+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13892409,"num_objects":1696,"num_object_clones":0,"num_object_copies":6784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10202,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10381","reported_seq":14557,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:08.171711+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:08.171711+0000","last_peered":"2024-09-15T03:15:08.171711+0000","last_clean":"2024-09-15T03:15:08.171711+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:08.171711+0000","last_undegraded":"2024-09-15T03:15:08.171711+0000","last_fullsized":"2024-09-15T03:15:08.171711+0000","mapping_epoch":36,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14351161,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10380,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10019","reported_seq":10121,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.329761+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:05.329761+0000","last_peered":"2024-09-15T03:15:05.329761+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:05.329761+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13620850,"num_objects":1663,"num_object_clones":0,"num_object_copies":6652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10017,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-350}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10186","reported_seq":10295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:05.609547+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:05.609547+0000","last_peered":"2024-09-15T03:15:05.609547+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:05.609547+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13899378,"num_objects":1697,"num_object_clones":0,"num_object_copies":6788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10184,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-280}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10278","reported_seq":10393,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:04.914641+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:04.914641+0000","last_peered":"2024-09-15T03:15:04.914641+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:04.914641+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14177906,"num_objects":1731,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10276,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-328}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":223967828,"num_objects":27345,"num_object_clones":0,"num_object_copies":109380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27345,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163362,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6279,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":447442944,"data_stored":447442944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160489,"ondisk_log_size":160489,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700408,"kb_used_data":110000,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93671432,"statfs":{"total":96636764160,"available":95919546368,"internally_reserved":0,"allocated":112640000,"data_stored":112456179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.8520000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.33500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.30399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704100,"kb_used_data":110044,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93667740,"statfs":{"total":96636764160,"available":95915765760,"internally_reserved":0,"allocated":112685056,"data_stored":112504381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.87},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.52400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.48899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.70099999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419188,"kb_used_data":108844,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93952652,"statfs":{"total":96636764160,"available":96207515648,"internally_reserved":0,"allocated":111456256,"data_stored":111272435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":8.3369999999999997},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":9.0519999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":9.0229999999999997},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":9.1639999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":8.9969999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":8.9789999999999992}]}]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709396,"kb_used_data":110732,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93662444,"statfs":{"total":96636764160,"available":95910342656,"internally_reserved":0,"allocated":113389568,"data_stored":113205747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,3,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.19900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.23100000000000001},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.28899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":112459776,"data_stored":112459776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110575616,"data_stored":110575616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":112119808,"data_stored":112119808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":112287744,"data_stored":112287744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:10.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10195", 'reported_seq': 10262, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.435307+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:15:05.435307+0000', 'last_peered': '2024-09-15T03:15:05.435307+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:15:05.435307+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14049280, 'num_objects': 1715, 'num_object_clones': 0, 'num_object_copies': 6860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10195, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -251}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9818", 'reported_seq': 10001, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.235108+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:15:05.235108+0000', 'last_peered': '2024-09-15T03:15:05.235108+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:15:05.235108+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9818, 'log_dups_size': 0, 'ondisk_log_size': 9818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13170290, 'num_objects': 1608, 'num_object_clones': 0, 'num_object_copies': 6432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1943, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9816, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 59, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10441", 'reported_seq': 10551, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.324466+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:15:05.324466+0000', 'last_peered': '2024-09-15T03:15:05.324466+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:15:05.324466+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14572369, 'num_objects': 1780, 'num_object_clones': 0, 'num_object_copies': 7120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1780, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10440, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -311}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10341", 'reported_seq': 10449, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.406475+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:15:05.406475+0000', 'last_peered': '2024-09-15T03:15:05.406475+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:15:05.406475+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14251634, 'num_objects': 1740, 'num_object_clones': 0, 'num_object_copies': 6960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10339, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -344}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:15:10.996 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10299", 'reported_seq': 10385, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.302777+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:15:05.302777+0000', 'last_peered': '2024-09-15T03:15:05.302777+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:15:05.302777+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13918208, 'num_objects': 1699, 'num_object_clones': 0, 'num_object_copies': 6796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1699, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10299, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -339}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:10.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:15:10.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10252", 'reported_seq': 10375, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.009044+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:15:05.009044+0000', 'last_peered': '2024-09-15T03:15:05.009044+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:15:05.009044+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13973106, 'num_objects': 1706, 'num_object_clones': 0, 'num_object_copies': 6824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10250, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -320}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:10.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:15:10.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10343", 'reported_seq': 10462, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.179640+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:15:05.179640+0000', 'last_peered': '2024-09-15T03:15:05.179640+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:15:05.179640+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14268018, 'num_objects': 1742, 'num_object_clones': 0, 'num_object_copies': 6968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10341, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:10.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:15:10.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10229", 'reported_seq': 10322, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.405109+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:15:05.405109+0000', 'last_peered': '2024-09-15T03:15:05.405109+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:15:05.405109+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14257380, 'num_objects': 1741, 'num_object_clones': 0, 'num_object_copies': 6964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10226, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -286}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:10.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:15:10.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10231", 'reported_seq': 10352, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.673726+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:15:05.673726+0000', 'last_peered': '2024-09-15T03:15:05.673726+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:15:05.673726+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14227058, 'num_objects': 1737, 'num_object_clones': 0, 'num_object_copies': 6948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10229, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -305}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:10.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:15:10.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10215", 'reported_seq': 10311, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:04.962588+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:15:04.962588+0000', 'last_peered': '2024-09-15T03:15:04.962588+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:15:04.962588+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13660587, 'num_objects': 1668, 'num_object_clones': 0, 'num_object_copies': 6672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10212, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:10.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:15:10.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10019", 'reported_seq': 10121, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.329761+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:15:05.329761+0000', 'last_peered': '2024-09-15T03:15:05.329761+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:15:05.329761+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13620850, 'num_objects': 1663, 'num_object_clones': 0, 'num_object_copies': 6652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10017, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -350}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:10.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:15:10.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10186", 'reported_seq': 10295, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:05.609547+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:15:05.609547+0000', 'last_peered': '2024-09-15T03:15:05.609547+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:15:05.609547+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13899378, 'num_objects': 1697, 'num_object_clones': 0, 'num_object_copies': 6788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10184, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:10.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:15:10.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10278", 'reported_seq': 10393, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:04.914641+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:15:04.914641+0000', 'last_peered': '2024-09-15T03:15:04.914641+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:15:04.914641+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14177906, 'num_objects': 1731, 'num_object_clones': 0, 'num_object_copies': 6924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10276, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -328}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:10.999 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:15:13.200 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:13.200 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26355/106580 objects degraded (24.728%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":636,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26645,"data_bytes":218233428,"bytes_used":2579570688,"bytes_avail":383967485952,"bytes_total":386547056640,"degraded_objects":26355,"degraded_total":106580,"degraded_ratio":0.24727903921936573,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":150},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:15:13.219 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:15:13.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:14.805 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:14.828 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:14.876 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-15T03:15:12.918409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218233428,"num_objects":26645,"num_object_clones":0,"num_object_copies":106580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26645,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164062,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6329,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160689,"ondisk_log_size":160689,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2519112,"kb_used_data":429784,"kb_used_omap":14,"kb_used_meta":2089265,"kb_avail":374968248,"statfs":{"total":386547056640,"available":383967485952,"internally_reserved":0,"allocated":440098816,"data_stored":439366678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139407387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,10,13,10,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14557184,"num_objects":-1777,"num_object_clones":0,"num_object_copies":-7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.802344"},"pg_stats":[{"pgid":"1.f","version":"37'10242","reported_seq":10310,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.116088+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:10.116088+0000","last_peered":"2024-09-15T03:15:10.116088+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:10.116088+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13664256,"num_objects":1668,"num_object_clones":0,"num_object_copies":6672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10242,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9865","reported_seq":10148,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.280250+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:10.280250+0000","last_peered":"2024-09-15T03:15:10.280250+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:10.280250+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12785266,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9863,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10498","reported_seq":10608,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.540206+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:10.540206+0000","last_peered":"2024-09-15T03:15:10.540206+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:10.540206+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14105425,"num_objects":1723,"num_object_clones":0,"num_object_copies":6892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1723,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10497,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-368}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'9958","reported_seq":13932,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:08.156427+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:08.156427+0000","last_peered":"2024-09-15T03:15:08.156427+0000","last_clean":"2024-09-15T03:15:08.156427+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:08.156427+0000","last_undegraded":"2024-09-15T03:15:08.156427+0000","last_fullsized":"2024-09-15T03:15:08.156427+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9958,"log_dups_size":0,"ondisk_log_size":9958,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13678194,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9956,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10390","reported_seq":10498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.235628+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:10.235628+0000","last_peered":"2024-09-15T03:15:10.235628+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:10.235628+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13850226,"num_objects":1691,"num_object_clones":0,"num_object_copies":6764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10388,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10350","reported_seq":10436,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.383355+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:10.383355+0000","last_peered":"2024-09-15T03:15:10.383355+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:10.383355+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13500416,"num_objects":1648,"num_object_clones":0,"num_object_copies":6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10350,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-390}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10303","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.220683+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:10.220683+0000","last_peered":"2024-09-15T03:15:10.220683+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:10.220683+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13555314,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10301,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10399","reported_seq":10518,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.083668+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:10.083668+0000","last_peered":"2024-09-15T03:15:10.083668+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:10.083668+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13809266,"num_objects":1686,"num_object_clones":0,"num_object_copies":6744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10397,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-370}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10286","reported_seq":10379,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.492488+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:10.492488+0000","last_peered":"2024-09-15T03:15:10.492488+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:10.492488+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13790436,"num_objects":1684,"num_object_clones":0,"num_object_copies":6736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10283,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-343}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10288","reported_seq":10406,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.614054+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:10.614054+0000","last_peered":"2024-09-15T03:15:10.614054+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:10.614054+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13760114,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10286,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10262","reported_seq":10358,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.064136+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:10.064136+0000","last_peered":"2024-09-15T03:15:10.064136+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:10.064136+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13275563,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10259,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-441}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10203","reported_seq":14246,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:07.923082+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:07.923082+0000","last_peered":"2024-09-15T03:15:07.923082+0000","last_clean":"2024-09-15T03:15:07.923082+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:07.923082+0000","last_undegraded":"2024-09-15T03:15:07.923082+0000","last_fullsized":"2024-09-15T03:15:07.923082+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13892409,"num_objects":1696,"num_object_clones":0,"num_object_copies":6784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10202,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10381","reported_seq":14557,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:08.171711+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:08.171711+0000","last_peered":"2024-09-15T03:15:08.171711+0000","last_clean":"2024-09-15T03:15:08.171711+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:08.171711+0000","last_undegraded":"2024-09-15T03:15:08.171711+0000","last_fullsized":"2024-09-15T03:15:08.171711+0000","mapping_epoch":36,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14351161,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10380,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10076","reported_seq":10178,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.462622+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:10.462622+0000","last_peered":"2024-09-15T03:15:10.462622+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:10.462622+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13153906,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10244","reported_seq":10353,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.516965+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:10.516965+0000","last_peered":"2024-09-15T03:15:10.516965+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:10.516965+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13424242,"num_objects":1639,"num_object_clones":0,"num_object_copies":6556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10242,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-338}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10344","reported_seq":10459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:10.200949+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:10.200949+0000","last_peered":"2024-09-15T03:15:10.200949+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:10.200949+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13637234,"num_objects":1665,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10342,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-394}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":218233428,"num_objects":26645,"num_object_clones":0,"num_object_copies":106580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26645,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164062,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6329,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":436731904,"data_stored":436731904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160689,"ondisk_log_size":160689,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701724,"kb_used_data":107140,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93670116,"statfs":{"total":96636764160,"available":95918198784,"internally_reserved":0,"allocated":109711360,"data_stored":109527539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.19400000000000001},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":7.6669999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":3.7839999999999998},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.23100000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":3.9470000000000001},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.28799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704776,"kb_used_data":106560,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93667064,"statfs":{"total":96636764160,"available":95915073536,"internally_reserved":0,"allocated":109117440,"data_stored":108936765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,0,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":5.2510000000000003},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":5.1959999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":15.125999999999999},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":15.170999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.378},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.316}]}]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419188,"kb_used_data":108844,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93952652,"statfs":{"total":96636764160,"available":96207515648,"internally_reserved":0,"allocated":111456256,"data_stored":111272435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":7.1559999999999997,"5min":2.6509999999999998,"15min":2.1720000000000002},"min":{"1min":0.24399999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.076000000000001,"5min":28.076000000000001,"15min":28.076000000000001},"last":8.3369999999999997},{"interface":"front","average":{"1min":3.5129999999999999,"5min":2.3809999999999998,"15min":1.8009999999999999},"min":{"1min":0.20499999999999999,"5min":0.191,"15min":0.187},"max":{"1min":49.399999999999999,"5min":49.399999999999999,"15min":49.399999999999999},"last":9.0519999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.3019999999999996,"5min":2.7519999999999998,"15min":1.8340000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":25.184999999999999,"5min":26.911000000000001,"15min":26.911000000000001},"last":9.0229999999999997},{"interface":"front","average":{"1min":9.1280000000000001,"5min":4.2359999999999998,"15min":2.464},"min":{"1min":0.25900000000000001,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":28.145,"5min":28.145,"15min":28.145},"last":9.1639999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:14:10 2024","interfaces":[{"interface":"back","average":{"1min":5.194,"5min":2.7000000000000002,"15min":1.927},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":59.225999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":8.9969999999999999},{"interface":"front","average":{"1min":1.526,"5min":1.966,"15min":1.52},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":12.102,"5min":20.681000000000001,"15min":20.681000000000001},"last":8.9789999999999992}]}]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693424,"kb_used_data":107240,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93678416,"statfs":{"total":96636764160,"available":95926697984,"internally_reserved":0,"allocated":109813760,"data_stored":109629939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,5,3,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":8.4740000000000002},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":8.6600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":1.399},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.5}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.30299999999999999},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.34599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":108843008,"data_stored":108843008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110575616,"data_stored":110575616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":108580864,"data_stored":108580864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":108732416,"data_stored":108732416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:17.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:20.195 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:20.292 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:20.325 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-15T03:15:18.920402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":209328724,"num_objects":25558,"num_object_clones":0,"num_object_copies":102232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165149,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6375,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2491876,"kb_used_data":410852,"kb_used_omap":14,"kb_used_meta":2080945,"kb_avail":374995484,"statfs":{"total":386547056640,"available":383995375616,"internally_reserved":0,"allocated":420712448,"data_stored":419980310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130887707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,2,11,10,16,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16375808,"num_objects":-1999,"num_object_clones":0,"num_object_copies":-7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1999,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"1.f","version":"37'10294","reported_seq":10362,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.316815+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:15.316815+0000","last_peered":"2024-09-15T03:15:15.316815+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:15.316815+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13238272,"num_objects":1616,"num_object_clones":0,"num_object_copies":6464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10294,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9921","reported_seq":10296,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.193612+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:15.193612+0000","last_peered":"2024-09-15T03:15:15.193612+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:15.193612+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12326514,"num_objects":1505,"num_object_clones":0,"num_object_copies":6020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10541","reported_seq":10651,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.630905+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:15.630905+0000","last_peered":"2024-09-15T03:15:15.630905+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:15.630905+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13753169,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10540,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-411}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10086","reported_seq":14057,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:18.258120+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:18.258120+0000","last_peered":"2024-09-15T03:15:18.258120+0000","last_clean":"2024-09-15T03:15:18.258120+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:18.258120+0000","last_undegraded":"2024-09-15T03:15:18.258120+0000","last_fullsized":"2024-09-15T03:15:18.258120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12629618,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10084,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10446","reported_seq":10554,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.464051+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:15.464051+0000","last_peered":"2024-09-15T03:15:15.464051+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:15.464051+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13391474,"num_objects":1635,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10444,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10410","reported_seq":10496,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.411853+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:15.411853+0000","last_peered":"2024-09-15T03:15:15.411853+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:15.411853+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13008896,"num_objects":1588,"num_object_clones":0,"num_object_copies":6352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10410,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-450}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10360","reported_seq":10483,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.395428+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:15.395428+0000","last_peered":"2024-09-15T03:15:15.395428+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:15.395428+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13088370,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10358,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-428}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10466","reported_seq":10585,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.258503+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:15.258503+0000","last_peered":"2024-09-15T03:15:15.258503+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:15.258503+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13260402,"num_objects":1619,"num_object_clones":0,"num_object_copies":6476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-437}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10332","reported_seq":10425,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.534716+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:15.534716+0000","last_peered":"2024-09-15T03:15:15.534716+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:15.534716+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13413604,"num_objects":1638,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10329,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10344","reported_seq":10465,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.597885+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:15.597885+0000","last_peered":"2024-09-15T03:15:15.597885+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:15.597885+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13301362,"num_objects":1624,"num_object_clones":0,"num_object_copies":6496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10342,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10319","reported_seq":10415,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.436790+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:15.436790+0000","last_peered":"2024-09-15T03:15:15.436790+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:15.436790+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12808619,"num_objects":1564,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10317","reported_seq":14360,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:18.398482+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:18.398482+0000","last_peered":"2024-09-15T03:15:18.398482+0000","last_clean":"2024-09-15T03:15:18.398482+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:18.398482+0000","last_undegraded":"2024-09-15T03:15:18.398482+0000","last_fullsized":"2024-09-15T03:15:18.398482+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12958521,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10499","reported_seq":14675,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:18.307150+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:18.307150+0000","last_peered":"2024-09-15T03:15:18.307150+0000","last_clean":"2024-09-15T03:15:18.307150+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:18.307150+0000","last_undegraded":"2024-09-15T03:15:18.307150+0000","last_fullsized":"2024-09-15T03:15:18.307150+0000","mapping_epoch":36,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13384505,"num_objects":1634,"num_object_clones":0,"num_object_copies":6536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10134","reported_seq":10236,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.466814+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:15.466814+0000","last_peered":"2024-09-15T03:15:15.466814+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:15.466814+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12678770,"num_objects":1548,"num_object_clones":0,"num_object_copies":6192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10132,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-465}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10313","reported_seq":10418,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.680107+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:15.680107+0000","last_peered":"2024-09-15T03:15:15.680107+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:15.680107+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12858994,"num_objects":1570,"num_object_clones":0,"num_object_copies":6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-407}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10394","reported_seq":10509,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.326795+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:15.326795+0000","last_peered":"2024-09-15T03:15:15.326795+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:15.326795+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13227634,"num_objects":1615,"num_object_clones":0,"num_object_copies":6460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":209328724,"num_objects":25558,"num_object_clones":0,"num_object_copies":102232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26309,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165149,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6375,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":418316288,"data_stored":418316288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701664,"kb_used_data":102924,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93670176,"statfs":{"total":96636764160,"available":95918260224,"internally_reserved":0,"allocated":105394176,"data_stored":105210355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":16.449000000000002},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":6.125}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.49299999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.45900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688864,"kb_used_data":103108,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93682976,"statfs":{"total":96636764160,"available":95931367424,"internally_reserved":0,"allocated":105582592,"data_stored":105401917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":4.2999999999999998},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.218}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.39200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.42799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407252,"kb_used_data":101068,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93964588,"statfs":{"total":96636764160,"available":96219738112,"internally_reserved":0,"allocated":103493632,"data_stored":103309811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":15.23},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.22800000000000001},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":7.25}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":3.5699999999999998},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.39000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694096,"kb_used_data":103752,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93677744,"statfs":{"total":96636764160,"available":95926009856,"internally_reserved":0,"allocated":106242048,"data_stored":106058227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,6,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":0.46800000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.436},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.35899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.29299999999999998},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.51900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":105177088,"data_stored":105177088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103141376,"data_stored":103141376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104955904,"data_stored":104955904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":105041920,"data_stored":105041920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:20.326 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:21.775 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:21.781 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:21.791 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-15T03:15:20.920857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":204692052,"num_objects":24992,"num_object_clones":0,"num_object_copies":99968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165715,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6415,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160742,"ondisk_log_size":160742,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2475712,"kb_used_data":403020,"kb_used_omap":14,"kb_used_meta":2072625,"kb_avail":375011648,"statfs":{"total":386547056640,"available":384011927552,"internally_reserved":0,"allocated":412692480,"data_stored":411960342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2122368027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,2,3,5,19,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19275776,"num_objects":-2353,"num_object_clones":0,"num_object_copies":-9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2353,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"1.f","version":"37'10358","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.528921+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:20.528921+0000","last_peered":"2024-09-15T03:15:20.528921+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:20.528921+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12713984,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10358,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9967","reported_seq":10422,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.344437+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:20.344437+0000","last_peered":"2024-09-15T03:15:20.344437+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:20.344437+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11949682,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9965,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10541","reported_seq":10651,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.630905+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:15.630905+0000","last_peered":"2024-09-15T03:15:15.630905+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:15.630905+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13753169,"num_objects":1680,"num_object_clones":0,"num_object_copies":6720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10540,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-411}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10086","reported_seq":14057,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:18.258120+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:18.258120+0000","last_peered":"2024-09-15T03:15:18.258120+0000","last_clean":"2024-09-15T03:15:18.258120+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:18.258120+0000","last_undegraded":"2024-09-15T03:15:18.258120+0000","last_fullsized":"2024-09-15T03:15:18.258120+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12629618,"num_objects":1542,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10084,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10505","reported_seq":10613,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.438614+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:20.438614+0000","last_peered":"2024-09-15T03:15:20.438614+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:20.438614+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12908146,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10473","reported_seq":10559,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.639451+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:20.639451+0000","last_peered":"2024-09-15T03:15:20.639451+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:20.639451+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12492800,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10418","reported_seq":10541,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.242154+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:20.242154+0000","last_peered":"2024-09-15T03:15:20.242154+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:20.242154+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12613234,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10416,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10519","reported_seq":10638,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.180659+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:20.180659+0000","last_peered":"2024-09-15T03:15:20.180659+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:20.180659+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12826226,"num_objects":1566,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10517,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10385","reported_seq":10478,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.578157+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:20.578157+0000","last_peered":"2024-09-15T03:15:20.578157+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:20.578157+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12979428,"num_objects":1585,"num_object_clones":0,"num_object_copies":6340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10382,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10344","reported_seq":10465,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.597885+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:15.597885+0000","last_peered":"2024-09-15T03:15:15.597885+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:15.597885+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13301362,"num_objects":1624,"num_object_clones":0,"num_object_copies":6496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10342,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-418}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10380","reported_seq":10476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.467348+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:20.467348+0000","last_peered":"2024-09-15T03:15:20.467348+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:20.467348+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12308907,"num_objects":1503,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10377,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10317","reported_seq":14360,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:18.398482+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:18.398482+0000","last_peered":"2024-09-15T03:15:18.398482+0000","last_clean":"2024-09-15T03:15:18.398482+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:18.398482+0000","last_undegraded":"2024-09-15T03:15:18.398482+0000","last_fullsized":"2024-09-15T03:15:18.398482+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12958521,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10316,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10499","reported_seq":14675,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:18.307150+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:18.307150+0000","last_peered":"2024-09-15T03:15:18.307150+0000","last_clean":"2024-09-15T03:15:18.307150+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:18.307150+0000","last_undegraded":"2024-09-15T03:15:18.307150+0000","last_fullsized":"2024-09-15T03:15:18.307150+0000","mapping_epoch":36,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13384505,"num_objects":1634,"num_object_clones":0,"num_object_copies":6536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10180","reported_seq":10282,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.356625+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:20.356625+0000","last_peered":"2024-09-15T03:15:20.356625+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:20.356625+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12301938,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10178,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-511}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10313","reported_seq":10418,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:15.680107+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:15.680107+0000","last_peered":"2024-09-15T03:15:15.680107+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:15.680107+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12858994,"num_objects":1570,"num_object_clones":0,"num_object_copies":6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-407}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10457","reported_seq":10572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.215170+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:20.215170+0000","last_peered":"2024-09-15T03:15:20.215170+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:20.215170+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12711538,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10455,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":204692052,"num_objects":24992,"num_object_clones":0,"num_object_copies":99968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165715,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6415,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":410947584,"data_stored":410947584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160742,"ondisk_log_size":160742,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685632,"kb_used_data":99384,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93686208,"statfs":{"total":96636764160,"available":95934676992,"internally_reserved":0,"allocated":101769216,"data_stored":101585395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":1.722},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.76}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.47099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.307},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.36399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688864,"kb_used_data":103108,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93682976,"statfs":{"total":96636764160,"available":95931367424,"internally_reserved":0,"allocated":105582592,"data_stored":105401917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":4.2999999999999998},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.218}]},{"osd":1,"last update":"Sun Sep 15 03:14:18 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.8939999999999999,"15min":1.4099999999999999},"min":{"1min":0.23000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":7.4669999999999996,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.903,"5min":1.9359999999999999,"15min":1.466},"min":{"1min":0.31,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":7.5119999999999996,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.39200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.42799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407252,"kb_used_data":101068,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93964588,"statfs":{"total":96636764160,"available":96219738112,"internally_reserved":0,"allocated":103493632,"data_stored":103309811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":15.23},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.22800000000000001},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":7.25}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":3.5699999999999998},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.39000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693964,"kb_used_data":99460,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93677876,"statfs":{"total":96636764160,"available":95926145024,"internally_reserved":0,"allocated":101847040,"data_stored":101663219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.4859999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":6.4459999999999997},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.497}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.36399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101490688,"data_stored":101490688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103141376,"data_stored":103141376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104955904,"data_stored":104955904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":101359616,"data_stored":101359616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:21.792 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:15:21.792 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10358", 'reported_seq': 10426, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.528921+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:15:20.528921+0000', 'last_peered': '2024-09-15T03:15:20.528921+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:15:20.528921+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12713984, 'num_objects': 1552, 'num_object_clones': 0, 'num_object_copies': 6208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10358, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -414}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'9967", 'reported_seq': 10422, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.344437+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:15:20.344437+0000', 'last_peered': '2024-09-15T03:15:20.344437+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:15:20.344437+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 9967, 'log_dups_size': 0, 'ondisk_log_size': 9967, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11949682, 'num_objects': 1459, 'num_object_clones': 0, 'num_object_copies': 5836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1807, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9965, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 195, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10541", 'reported_seq': 10651, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:15.630905+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:15:15.630905+0000', 'last_peered': '2024-09-15T03:15:15.630905+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:15:15.630905+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13753169, 'num_objects': 1680, 'num_object_clones': 0, 'num_object_copies': 6720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1680, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10540, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -411}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10505", 'reported_seq': 10613, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.438614+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:15:20.438614+0000', 'last_peered': '2024-09-15T03:15:20.438614+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:15:20.438614+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12908146, 'num_objects': 1576, 'num_object_clones': 0, 'num_object_copies': 6304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10503, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -508}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10473", 'reported_seq': 10559, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.639451+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:15:20.639451+0000', 'last_peered': '2024-09-15T03:15:20.639451+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:15:20.639451+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12492800, 'num_objects': 1525, 'num_object_clones': 0, 'num_object_copies': 6100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10473, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -513}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10418", 'reported_seq': 10541, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.242154+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:15:20.242154+0000', 'last_peered': '2024-09-15T03:15:20.242154+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:15:20.242154+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12613234, 'num_objects': 1540, 'num_object_clones': 0, 'num_object_copies': 6160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10416, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -486}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:21.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10519", 'reported_seq': 10638, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.180659+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:15:20.180659+0000', 'last_peered': '2024-09-15T03:15:20.180659+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:15:20.180659+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12826226, 'num_objects': 1566, 'num_object_clones': 0, 'num_object_copies': 6264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10517, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -490}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10385", 'reported_seq': 10478, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.578157+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:15:20.578157+0000', 'last_peered': '2024-09-15T03:15:20.578157+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:15:20.578157+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12979428, 'num_objects': 1585, 'num_object_clones': 0, 'num_object_copies': 6340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10382, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -442}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10344", 'reported_seq': 10465, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:15.597885+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:15:15.597885+0000', 'last_peered': '2024-09-15T03:15:15.597885+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:15:15.597885+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13301362, 'num_objects': 1624, 'num_object_clones': 0, 'num_object_copies': 6496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10342, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -418}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10380", 'reported_seq': 10476, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.467348+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:15:20.467348+0000', 'last_peered': '2024-09-15T03:15:20.467348+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:15:20.467348+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12308907, 'num_objects': 1503, 'num_object_clones': 0, 'num_object_copies': 6012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10377, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -559}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10180", 'reported_seq': 10282, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.356625+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:15:20.356625+0000', 'last_peered': '2024-09-15T03:15:20.356625+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:15:20.356625+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12301938, 'num_objects': 1502, 'num_object_clones': 0, 'num_object_copies': 6008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10178, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:15:21.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10313", 'reported_seq': 10418, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:15.680107+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:15:15.680107+0000', 'last_peered': '2024-09-15T03:15:15.680107+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:15:15.680107+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12858994, 'num_objects': 1570, 'num_object_clones': 0, 'num_object_copies': 6280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10311, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -407}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:21.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:15:21.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10457", 'reported_seq': 10572, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:20.215170+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:15:20.215170+0000', 'last_peered': '2024-09-15T03:15:20.215170+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:15:20.215170+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12711538, 'num_objects': 1552, 'num_object_clones': 0, 'num_object_copies': 6208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10455, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:21.795 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:15:23.652 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:23.652 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26269/99324 objects degraded (26.448%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":646,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24831,"data_bytes":203373140,"bytes_used":2535731200,"bytes_avail":384011325440,"bytes_total":386547056640,"degraded_objects":26269,"degraded_total":99324,"degraded_ratio":0.26447787040393056,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":151},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:15:23.740 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:15:23.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:25.702 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:25.764 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:25.772 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-15T03:15:24.924878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":201898580,"num_objects":24651,"num_object_clones":0,"num_object_copies":98604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166056,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6415,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2476908,"kb_used_data":395916,"kb_used_omap":14,"kb_used_meta":2080945,"kb_avail":375010452,"statfs":{"total":386547056640,"available":384010702848,"internally_reserved":0,"allocated":405417984,"data_stored":404685846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130887707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,4,7,11,17,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16334848,"num_objects":-1994,"num_object_clones":0,"num_object_copies":-7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1994,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006468"},"pg_stats":[{"pgid":"1.f","version":"37'10358","reported_seq":10426,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.528921+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:20.528921+0000","last_peered":"2024-09-15T03:15:20.528921+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:20.528921+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12713984,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10358,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'9967","reported_seq":10422,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.344437+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:20.344437+0000","last_peered":"2024-09-15T03:15:20.344437+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:20.344437+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11949682,"num_objects":1459,"num_object_clones":0,"num_object_copies":5836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9965,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10600","reported_seq":10709,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.790882+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:20.790882+0000","last_peered":"2024-09-15T03:15:20.790882+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:20.790882+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13269841,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10599,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-470}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10149","reported_seq":14123,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:23.326038+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:23.326038+0000","last_peered":"2024-09-15T03:15:23.326038+0000","last_clean":"2024-09-15T03:15:23.326038+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:23.326038+0000","last_undegraded":"2024-09-15T03:15:23.326038+0000","last_fullsized":"2024-09-15T03:15:23.326038+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12113522,"num_objects":1479,"num_object_clones":0,"num_object_copies":5916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10505","reported_seq":10613,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.438614+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:20.438614+0000","last_peered":"2024-09-15T03:15:20.438614+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:20.438614+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12908146,"num_objects":1576,"num_object_clones":0,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10473","reported_seq":10559,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.639451+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:20.639451+0000","last_peered":"2024-09-15T03:15:20.639451+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:20.639451+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12492800,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10418","reported_seq":10541,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.242154+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:20.242154+0000","last_peered":"2024-09-15T03:15:20.242154+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:20.242154+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12613234,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10416,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10519","reported_seq":10638,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.180659+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:20.180659+0000","last_peered":"2024-09-15T03:15:20.180659+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:20.180659+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12826226,"num_objects":1566,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10517,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10385","reported_seq":10478,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.578157+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:20.578157+0000","last_peered":"2024-09-15T03:15:20.578157+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:20.578157+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12979428,"num_objects":1585,"num_object_clones":0,"num_object_copies":6340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10382,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-442}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10393","reported_seq":10512,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.879545+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:20.879545+0000","last_peered":"2024-09-15T03:15:20.879545+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:20.879545+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12899954,"num_objects":1575,"num_object_clones":0,"num_object_copies":6300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10391,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-467}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10380","reported_seq":10476,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.467348+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:20.467348+0000","last_peered":"2024-09-15T03:15:20.467348+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:20.467348+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12308907,"num_objects":1503,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10377,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10386","reported_seq":14426,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:23.513821+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:23.513821+0000","last_peered":"2024-09-15T03:15:23.513821+0000","last_clean":"2024-09-15T03:15:23.513821+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:23.513821+0000","last_undegraded":"2024-09-15T03:15:23.513821+0000","last_fullsized":"2024-09-15T03:15:23.513821+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12393273,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10385,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10547","reported_seq":14723,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:23.406711+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:23.406711+0000","last_peered":"2024-09-15T03:15:23.406711+0000","last_clean":"2024-09-15T03:15:23.406711+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:23.406711+0000","last_undegraded":"2024-09-15T03:15:23.406711+0000","last_fullsized":"2024-09-15T03:15:23.406711+0000","mapping_epoch":36,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12991289,"num_objects":1586,"num_object_clones":0,"num_object_copies":6344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10546,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10180","reported_seq":10282,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.356625+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:20.356625+0000","last_peered":"2024-09-15T03:15:20.356625+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:20.356625+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12301938,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10178,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-511}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10366","reported_seq":10475,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.706817+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:20.706817+0000","last_peered":"2024-09-15T03:15:20.706817+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:20.706817+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12424818,"num_objects":1517,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10364,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-460}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10457","reported_seq":10572,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:20.215170+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:20.215170+0000","last_peered":"2024-09-15T03:15:20.215170+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:20.215170+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12711538,"num_objects":1552,"num_object_clones":0,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10455,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-507}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":201898580,"num_objects":24651,"num_object_clones":0,"num_object_copies":98604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166056,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6415,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":403492864,"data_stored":403492864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":685632,"kb_used_data":99384,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93686208,"statfs":{"total":96636764160,"available":95934676992,"internally_reserved":0,"allocated":101769216,"data_stored":101585395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":1.722},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.76}]},{"osd":1,"last update":"Sun Sep 15 03:14:22 2024","interfaces":[{"interface":"back","average":{"1min":3.5600000000000001,"5min":3.2410000000000001,"15min":2.2210000000000001},"min":{"1min":0.23499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":18.43,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":4.1609999999999996,"5min":2.2629999999999999,"15min":1.6240000000000001},"min":{"1min":0.30499999999999999,"5min":0.24099999999999999,"15min":0.20300000000000001},"max":{"1min":21.027999999999999,"5min":22.262,"15min":22.262},"last":0.47099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.307},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.36399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689452,"kb_used_data":99556,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93682388,"statfs":{"total":96636764160,"available":95930765312,"internally_reserved":0,"allocated":101945344,"data_stored":101764669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,4,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.48099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.499},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.63100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.52200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":407860,"kb_used_data":97516,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93963980,"statfs":{"total":96636764160,"available":96219115520,"internally_reserved":0,"allocated":99856384,"data_stored":99672563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.23300000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.245},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":0.309}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.26400000000000001},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.27600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693964,"kb_used_data":99460,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93677876,"statfs":{"total":96636764160,"available":95926145024,"internally_reserved":0,"allocated":101847040,"data_stored":101663219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.4859999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":6.4459999999999997},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.497}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.33500000000000002},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.36399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":101490688,"data_stored":101490688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99471360,"data_stored":99471360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101171200,"data_stored":101171200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":101359616,"data_stored":101359616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:28.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:30.635 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:30.648 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:30.670 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":355,"stamp":"2024-09-15T03:15:28.925841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194869844,"num_objects":23793,"num_object_clones":0,"num_object_copies":95172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23793,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166914,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6468,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2463408,"kb_used_data":382416,"kb_used_omap":14,"kb_used_meta":2080945,"kb_avail":375023952,"statfs":{"total":386547056640,"available":384024526848,"internally_reserved":0,"allocated":391593984,"data_stored":390861846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130887707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,9,10,6,11,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15982592,"num_objects":-1951,"num_object_clones":0,"num_object_copies":-7804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1951,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006511"},"pg_stats":[{"pgid":"1.f","version":"37'10406","reported_seq":10474,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.546634+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:25.546634+0000","last_peered":"2024-09-15T03:15:25.546634+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:25.546634+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":1504,"num_object_clones":0,"num_object_copies":6016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10406,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-462}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10016","reported_seq":10576,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.569502+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:25.569502+0000","last_peered":"2024-09-15T03:15:25.569502+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:25.569502+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11548274,"num_objects":1410,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10014,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10660","reported_seq":10769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.689560+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:25.689560+0000","last_peered":"2024-09-15T03:15:25.689560+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:25.689560+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12778321,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10659,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10185","reported_seq":14159,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:28.379087+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:28.379087+0000","last_peered":"2024-09-15T03:15:28.379087+0000","last_clean":"2024-09-15T03:15:28.379087+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:28.379087+0000","last_undegraded":"2024-09-15T03:15:28.379087+0000","last_fullsized":"2024-09-15T03:15:28.379087+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11818610,"num_objects":1443,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10555","reported_seq":10663,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.455250+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:25.455250+0000","last_peered":"2024-09-15T03:15:25.455250+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:25.455250+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12498546,"num_objects":1526,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10553,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10524","reported_seq":10610,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.684730+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:25.684730+0000","last_peered":"2024-09-15T03:15:25.684730+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:25.684730+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12075008,"num_objects":1474,"num_object_clones":0,"num_object_copies":5896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10524,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-564}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10476","reported_seq":10599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.300579+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:25.300579+0000","last_peered":"2024-09-15T03:15:25.300579+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:25.300579+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12138098,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10474,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-544}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10572","reported_seq":10689,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.496606+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:25.496606+0000","last_peered":"2024-09-15T03:15:25.496606+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:25.496606+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12392050,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10570,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-543}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10453","reported_seq":10546,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.694195+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:25.694195+0000","last_peered":"2024-09-15T03:15:25.694195+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:25.694195+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12422372,"num_objects":1517,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10450,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-510}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10447","reported_seq":10568,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.776713+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:25.776713+0000","last_peered":"2024-09-15T03:15:25.776713+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:25.776713+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457586,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10436","reported_seq":10532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.498343+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:25.498343+0000","last_peered":"2024-09-15T03:15:25.498343+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:25.498343+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11850155,"num_objects":1447,"num_object_clones":0,"num_object_copies":5788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10433,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10451","reported_seq":14494,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:28.274845+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:28.274845+0000","last_peered":"2024-09-15T03:15:28.274845+0000","last_clean":"2024-09-15T03:15:28.274845+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:28.274845+0000","last_undegraded":"2024-09-15T03:15:28.274845+0000","last_fullsized":"2024-09-15T03:15:28.274845+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11860793,"num_objects":1448,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10450,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10600","reported_seq":14776,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:28.571020+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:28.571020+0000","last_peered":"2024-09-15T03:15:28.571020+0000","last_clean":"2024-09-15T03:15:28.571020+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:28.571020+0000","last_undegraded":"2024-09-15T03:15:28.571020+0000","last_fullsized":"2024-09-15T03:15:28.571020+0000","mapping_epoch":36,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12557113,"num_objects":1533,"num_object_clones":0,"num_object_copies":6132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10599,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10231","reported_seq":10333,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.463557+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:25.463557+0000","last_peered":"2024-09-15T03:15:25.463557+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:25.463557+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11884146,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10229,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-562}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10407","reported_seq":10516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.445174+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:25.445174+0000","last_peered":"2024-09-15T03:15:25.445174+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:25.445174+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12088946,"num_objects":1476,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10405,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10522","reported_seq":10637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.348439+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:25.348439+0000","last_peered":"2024-09-15T03:15:25.348439+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:25.348439+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12179058,"num_objects":1487,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10520,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":194869844,"num_objects":23793,"num_object_clones":0,"num_object_copies":95172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23793,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166914,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6468,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":388943872,"data_stored":388943872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686260,"kb_used_data":95852,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93685580,"statfs":{"total":96636764160,"available":95934033920,"internally_reserved":0,"allocated":98152448,"data_stored":97968627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,1,4,2,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.79500000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.20699999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.32400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.29999999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690104,"kb_used_data":96048,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93681736,"statfs":{"total":96636764160,"available":95930097664,"internally_reserved":0,"allocated":98353152,"data_stored":98172477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,0,2,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":3.8199999999999998},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.21299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.434}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":3.4580000000000002},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":3.4969999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392416,"kb_used_data":94552,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93979424,"statfs":{"total":96636764160,"available":96234930176,"internally_reserved":0,"allocated":96821248,"data_stored":96637427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.28699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":7.4889999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.438},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.61699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694628,"kb_used_data":95964,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93677212,"statfs":{"total":96636764160,"available":95925465088,"internally_reserved":0,"allocated":98267136,"data_stored":98083315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:14:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.907,"15min":1.278},"min":{"1min":0.19500000000000001,"5min":0.188,"15min":0.17499999999999999},"max":{"1min":6.5499999999999998,"5min":18.594000000000001,"15min":19.527000000000001},"last":10.191000000000001},{"interface":"front","average":{"1min":1.6439999999999999,"5min":2.9950000000000001,"15min":2.149},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":11.74,"5min":30.266999999999999,"15min":30.266999999999999},"last":6.1710000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":6.202},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.25}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":1.585},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.315}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97820672,"data_stored":97820672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95772672,"data_stored":95772672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97595392,"data_stored":97595392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":97755136,"data_stored":97755136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:30.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:32.270 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:32.325 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:32.381 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-15T03:15:30.926355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190249556,"num_objects":23229,"num_object_clones":0,"num_object_copies":92916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167478,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6517,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161005,"ondisk_log_size":161005,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2448088,"kb_used_data":375416,"kb_used_omap":14,"kb_used_meta":2072625,"kb_avail":375039272,"statfs":{"total":386547056640,"available":384040214528,"internally_reserved":0,"allocated":384425984,"data_stored":383693846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2122368027},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,7,12,4,12,14],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19079168,"num_objects":-2329,"num_object_clones":0,"num_object_copies":-9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2329,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005953"},"pg_stats":[{"pgid":"1.f","version":"37'10469","reported_seq":10537,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.541393+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:30.541393+0000","last_peered":"2024-09-15T03:15:30.541393+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:30.541393+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11804672,"num_objects":1441,"num_object_clones":0,"num_object_copies":5764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10071","reported_seq":10730,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.648015+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:30.648015+0000","last_peered":"2024-09-15T03:15:30.648015+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:30.648015+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11097714,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10660","reported_seq":10769,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.689560+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:25.689560+0000","last_peered":"2024-09-15T03:15:25.689560+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:25.689560+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12778321,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10659,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-530}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10185","reported_seq":14159,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:28.379087+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:28.379087+0000","last_peered":"2024-09-15T03:15:28.379087+0000","last_clean":"2024-09-15T03:15:28.379087+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:28.379087+0000","last_undegraded":"2024-09-15T03:15:28.379087+0000","last_fullsized":"2024-09-15T03:15:28.379087+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11818610,"num_objects":1443,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10600","reported_seq":10708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.360629+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:30.360629+0000","last_peered":"2024-09-15T03:15:30.360629+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:30.360629+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12129906,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10598,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10591","reported_seq":10677,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.692418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:30.692418+0000","last_peered":"2024-09-15T03:15:30.692418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:30.692418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11526144,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-631}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10533","reported_seq":10656,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.537834+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:30.537834+0000","last_peered":"2024-09-15T03:15:30.537834+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:30.537834+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11671154,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-601}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10634","reported_seq":10752,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.443523+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:30.443523+0000","last_peered":"2024-09-15T03:15:30.443523+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:30.443523+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11884146,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10506","reported_seq":10599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.653682+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:30.653682+0000","last_peered":"2024-09-15T03:15:30.653682+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:30.653682+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11988196,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10447","reported_seq":10568,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.776713+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:25.776713+0000","last_peered":"2024-09-15T03:15:25.776713+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:25.776713+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12457586,"num_objects":1521,"num_object_clones":0,"num_object_copies":6084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10496","reported_seq":10592,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.562370+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:30.562370+0000","last_peered":"2024-09-15T03:15:30.562370+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:30.562370+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11358635,"num_objects":1387,"num_object_clones":0,"num_object_copies":5548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-675}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10451","reported_seq":14494,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:28.274845+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:28.274845+0000","last_peered":"2024-09-15T03:15:28.274845+0000","last_clean":"2024-09-15T03:15:28.274845+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:28.274845+0000","last_undegraded":"2024-09-15T03:15:28.274845+0000","last_fullsized":"2024-09-15T03:15:28.274845+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11860793,"num_objects":1448,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10450,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10600","reported_seq":14776,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:28.571020+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:28.571020+0000","last_peered":"2024-09-15T03:15:28.571020+0000","last_clean":"2024-09-15T03:15:28.571020+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:28.571020+0000","last_undegraded":"2024-09-15T03:15:28.571020+0000","last_fullsized":"2024-09-15T03:15:28.571020+0000","mapping_epoch":36,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12557113,"num_objects":1533,"num_object_clones":0,"num_object_copies":6132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10599,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10285","reported_seq":10387,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.522863+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:30.522863+0000","last_peered":"2024-09-15T03:15:30.522863+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:30.522863+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11441778,"num_objects":1397,"num_object_clones":0,"num_object_copies":5588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10283,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10407","reported_seq":10516,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:25.445174+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:25.445174+0000","last_peered":"2024-09-15T03:15:25.445174+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:25.445174+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12088946,"num_objects":1476,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10405,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10570","reported_seq":10685,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.584038+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:30.584038+0000","last_peered":"2024-09-15T03:15:30.584038+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:30.584038+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11785842,"num_objects":1439,"num_object_clones":0,"num_object_copies":5756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10568,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190249556,"num_objects":23229,"num_object_clones":0,"num_object_copies":92916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23229,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167478,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6517,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381698048,"data_stored":381698048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161005,"ondisk_log_size":161005,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686932,"kb_used_data":92364,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93684908,"statfs":{"total":96636764160,"available":95933345792,"internally_reserved":0,"allocated":94580736,"data_stored":94396915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.22600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.80000000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.51000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690104,"kb_used_data":96048,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93681736,"statfs":{"total":96636764160,"available":95930097664,"internally_reserved":0,"allocated":98353152,"data_stored":98172477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,1,0,2,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":3.8199999999999998},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.21299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.434}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":3.4580000000000002},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":3.4969999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392416,"kb_used_data":94552,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93979424,"statfs":{"total":96636764160,"available":96234930176,"internally_reserved":0,"allocated":96821248,"data_stored":96637427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.28699999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":7.4889999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.438},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.61699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678636,"kb_used_data":92452,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93693204,"statfs":{"total":96636764160,"available":95941840896,"internally_reserved":0,"allocated":94670848,"data_stored":94487027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,8,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":10.818}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":10.879},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.23599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94187520,"data_stored":94187520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95772672,"data_stored":95772672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97595392,"data_stored":97595392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94142464,"data_stored":94142464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:32.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10469", 'reported_seq': 10537, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.541393+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:15:30.541393+0000', 'last_peered': '2024-09-15T03:15:30.541393+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:15:30.541393+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11804672, 'num_objects': 1441, 'num_object_clones': 0, 'num_object_copies': 5764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10469, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -525}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10071", 'reported_seq': 10730, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.648015+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:15:30.648015+0000', 'last_peered': '2024-09-15T03:15:30.648015+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:15:30.648015+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11097714, 'num_objects': 1355, 'num_object_clones': 0, 'num_object_copies': 5420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1705, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10069, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 297, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -350}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10660", 'reported_seq': 10769, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:25.689560+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:15:25.689560+0000', 'last_peered': '2024-09-15T03:15:25.689560+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:15:25.689560+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12778321, 'num_objects': 1561, 'num_object_clones': 0, 'num_object_copies': 6244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1561, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10659, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -530}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10600", 'reported_seq': 10708, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.360629+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:15:30.360629+0000', 'last_peered': '2024-09-15T03:15:30.360629+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:15:30.360629+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12129906, 'num_objects': 1481, 'num_object_clones': 0, 'num_object_copies': 5924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10598, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -603}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:15:32.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10591", 'reported_seq': 10677, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.692418+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:15:30.692418+0000', 'last_peered': '2024-09-15T03:15:30.692418+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:15:30.692418+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11526144, 'num_objects': 1407, 'num_object_clones': 0, 'num_object_copies': 5628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10591, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -631}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10533", 'reported_seq': 10656, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.537834+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:15:30.537834+0000', 'last_peered': '2024-09-15T03:15:30.537834+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:15:30.537834+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11671154, 'num_objects': 1425, 'num_object_clones': 0, 'num_object_copies': 5700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10531, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -601}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10634", 'reported_seq': 10752, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.443523+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:15:30.443523+0000', 'last_peered': '2024-09-15T03:15:30.443523+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:15:30.443523+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11884146, 'num_objects': 1451, 'num_object_clones': 0, 'num_object_copies': 5804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10632, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -605}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10506", 'reported_seq': 10599, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.653682+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:15:30.653682+0000', 'last_peered': '2024-09-15T03:15:30.653682+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:15:30.653682+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11988196, 'num_objects': 1464, 'num_object_clones': 0, 'num_object_copies': 5856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10503, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -563}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10447", 'reported_seq': 10568, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:25.776713+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:15:25.776713+0000', 'last_peered': '2024-09-15T03:15:25.776713+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:15:25.776713+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12457586, 'num_objects': 1521, 'num_object_clones': 0, 'num_object_copies': 6084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10445, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -521}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10496", 'reported_seq': 10592, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.562370+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:15:30.562370+0000', 'last_peered': '2024-09-15T03:15:30.562370+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:15:30.562370+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11358635, 'num_objects': 1387, 'num_object_clones': 0, 'num_object_copies': 5548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1387, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10493, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -675}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:32.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:15:32.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10285", 'reported_seq': 10387, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.522863+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:15:30.522863+0000', 'last_peered': '2024-09-15T03:15:30.522863+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:15:30.522863+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11441778, 'num_objects': 1397, 'num_object_clones': 0, 'num_object_copies': 5588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10283, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -616}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:32.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:15:32.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10407", 'reported_seq': 10516, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:25.445174+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:15:25.445174+0000', 'last_peered': '2024-09-15T03:15:25.445174+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:15:25.445174+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12088946, 'num_objects': 1476, 'num_object_clones': 0, 'num_object_copies': 5904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10405, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:32.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:15:32.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10570", 'reported_seq': 10685, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:30.584038+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:15:30.584038+0000', 'last_peered': '2024-09-15T03:15:30.584038+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:15:30.584038+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11785842, 'num_objects': 1439, 'num_object_clones': 0, 'num_object_copies': 5756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10568, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:32.385 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:15:34.572 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:34.572 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26167/92276 objects degraded (28.357%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":657,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23069,"data_bytes":188938836,"bytes_used":2511683584,"bytes_avail":384035373056,"bytes_total":386547056640,"degraded_objects":26167,"degraded_total":92276,"degraded_ratio":0.28357319346308901,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":160},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:15:34.648 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:15:34.648 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:36.767 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:36.784 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:36.813 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-15T03:15:34.927388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":187472468,"num_objects":22890,"num_object_clones":0,"num_object_copies":91560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167817,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6517,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2453508,"kb_used_data":368356,"kb_used_omap":14,"kb_used_meta":2085105,"kb_avail":375033852,"statfs":{"total":386547056640,"available":384034664448,"internally_reserved":0,"allocated":377196544,"data_stored":376464406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135147547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,8,16,9,11,7,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15900672,"num_objects":-1941,"num_object_clones":0,"num_object_copies":-7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002984"},"pg_stats":[{"pgid":"1.f","version":"37'10469","reported_seq":10537,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.541393+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:30.541393+0000","last_peered":"2024-09-15T03:15:30.541393+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:30.541393+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11804672,"num_objects":1441,"num_object_clones":0,"num_object_copies":5764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-525}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10071","reported_seq":10730,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.648015+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:30.648015+0000","last_peered":"2024-09-15T03:15:30.648015+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:30.648015+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11097714,"num_objects":1355,"num_object_clones":0,"num_object_copies":5420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1705,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10069,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10720","reported_seq":10830,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:31.036014+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:31.036014+0000","last_peered":"2024-09-15T03:15:31.036014+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:31.036014+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12286801,"num_objects":1501,"num_object_clones":0,"num_object_copies":6004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10719,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10242","reported_seq":14216,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:33.408366+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:33.408366+0000","last_peered":"2024-09-15T03:15:33.408366+0000","last_clean":"2024-09-15T03:15:33.408366+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:33.408366+0000","last_undegraded":"2024-09-15T03:15:33.408366+0000","last_fullsized":"2024-09-15T03:15:33.408366+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11351666,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10600","reported_seq":10708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.360629+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:30.360629+0000","last_peered":"2024-09-15T03:15:30.360629+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:30.360629+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12129906,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10598,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10591","reported_seq":10677,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.692418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:30.692418+0000","last_peered":"2024-09-15T03:15:30.692418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:30.692418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11526144,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10591,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-631}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10533","reported_seq":10656,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.537834+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:30.537834+0000","last_peered":"2024-09-15T03:15:30.537834+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:30.537834+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11671154,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-601}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10634","reported_seq":10752,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.443523+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:30.443523+0000","last_peered":"2024-09-15T03:15:30.443523+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:30.443523+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11884146,"num_objects":1451,"num_object_clones":0,"num_object_copies":5804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10506","reported_seq":10599,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.653682+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:30.653682+0000","last_peered":"2024-09-15T03:15:30.653682+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:30.653682+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11988196,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10503","reported_seq":10624,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.904593+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:30.904593+0000","last_peered":"2024-09-15T03:15:30.904593+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:30.904593+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11998834,"num_objects":1465,"num_object_clones":0,"num_object_copies":5860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10501,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-577}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10496","reported_seq":10592,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.562370+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:30.562370+0000","last_peered":"2024-09-15T03:15:30.562370+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:30.562370+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11358635,"num_objects":1387,"num_object_clones":0,"num_object_copies":5548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10493,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-675}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10505","reported_seq":14548,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:33.606488+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:33.606488+0000","last_peered":"2024-09-15T03:15:33.606488+0000","last_clean":"2024-09-15T03:15:33.606488+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:33.606488+0000","last_undegraded":"2024-09-15T03:15:33.606488+0000","last_fullsized":"2024-09-15T03:15:33.606488+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11418425,"num_objects":1394,"num_object_clones":0,"num_object_copies":5576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10668","reported_seq":14844,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:33.680035+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:33.680035+0000","last_peered":"2024-09-15T03:15:33.680035+0000","last_clean":"2024-09-15T03:15:33.680035+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:33.680035+0000","last_undegraded":"2024-09-15T03:15:33.680035+0000","last_fullsized":"2024-09-15T03:15:33.680035+0000","mapping_epoch":36,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12000057,"num_objects":1465,"num_object_clones":0,"num_object_copies":5860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10667,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10285","reported_seq":10387,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.522863+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:30.522863+0000","last_peered":"2024-09-15T03:15:30.522863+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:30.522863+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11441778,"num_objects":1397,"num_object_clones":0,"num_object_copies":5588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10283,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-616}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10451","reported_seq":10560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:31.033067+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:31.033067+0000","last_peered":"2024-09-15T03:15:31.033067+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:31.033067+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11728498,"num_objects":1432,"num_object_clones":0,"num_object_copies":5728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10449,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-545}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10570","reported_seq":10685,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:30.584038+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:30.584038+0000","last_peered":"2024-09-15T03:15:30.584038+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:30.584038+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11785842,"num_objects":1439,"num_object_clones":0,"num_object_copies":5756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10568,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":187472468,"num_objects":22890,"num_object_clones":0,"num_object_copies":91560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167817,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6517,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":374554624,"data_stored":374554624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686932,"kb_used_data":92364,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93684908,"statfs":{"total":96636764160,"available":95933345792,"internally_reserved":0,"allocated":94580736,"data_stored":94396915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.22600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.80000000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.51000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694832,"kb_used_data":92456,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93677008,"statfs":{"total":96636764160,"available":95925256192,"internally_reserved":0,"allocated":94674944,"data_stored":94494269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,4,3,1,2,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":2.2629999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":4.8710000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.988},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":4.0209999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":4.0529999999999999},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":4.9080000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":393108,"kb_used_data":91084,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93978732,"statfs":{"total":96636764160,"available":96234221568,"internally_reserved":0,"allocated":93270016,"data_stored":93086195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.27200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":0.437}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":14.901},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.48199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678636,"kb_used_data":92452,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93693204,"statfs":{"total":96636764160,"available":95941840896,"internally_reserved":0,"allocated":94670848,"data_stored":94487027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,8,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":10.818}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":10.879},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":0.23599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":94187520,"data_stored":94187520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":92282880,"data_stored":92282880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93941760,"data_stored":93941760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94142464,"data_stored":94142464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:39.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:41.454 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:41.492 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:41.554 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-15T03:15:40.928885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":175741524,"num_objects":21458,"num_object_clones":0,"num_object_copies":85832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21458,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169249,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6612,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160876,"ondisk_log_size":160876,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2435864,"kb_used_data":346552,"kb_used_omap":14,"kb_used_meta":2089265,"kb_avail":375051496,"statfs":{"total":386547056640,"available":384052731904,"internally_reserved":0,"allocated":354869248,"data_stored":354137110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2139407387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,7,18,16,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19128320,"num_objects":-2335,"num_object_clones":0,"num_object_copies":-9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2335,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003043"},"pg_stats":[{"pgid":"1.f","version":"37'10589","reported_seq":10657,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.768810+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:40.768810+0000","last_peered":"2024-09-15T03:15:40.768810+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:40.768810+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10821632,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10181","reported_seq":11030,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.767404+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:40.767404+0000","last_peered":"2024-09-15T03:15:40.767404+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:40.767404+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10196594,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10179,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10781","reported_seq":10891,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:35.917621+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:35.917621+0000","last_peered":"2024-09-15T03:15:35.917621+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:35.917621+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11787089,"num_objects":1440,"num_object_clones":0,"num_object_copies":5760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10780,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-651}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10305","reported_seq":14279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:38.771428+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:38.771428+0000","last_peered":"2024-09-15T03:15:38.771428+0000","last_clean":"2024-09-15T03:15:38.771428+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:38.771428+0000","last_undegraded":"2024-09-15T03:15:38.771428+0000","last_fullsized":"2024-09-15T03:15:38.771428+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10835570,"num_objects":1323,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10718","reported_seq":10826,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.715550+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:40.715550+0000","last_peered":"2024-09-15T03:15:40.715550+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:40.715550+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11163250,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10716,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10692","reported_seq":10778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.659293+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:40.659293+0000","last_peered":"2024-09-15T03:15:40.659293+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:40.659293+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10698752,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10642","reported_seq":10764,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.836067+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:40.836067+0000","last_peered":"2024-09-15T03:15:40.836067+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:40.836067+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10778226,"num_objects":1316,"num_object_clones":0,"num_object_copies":5264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10640,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-710}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10739","reported_seq":10858,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.686084+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:40.686084+0000","last_peered":"2024-09-15T03:15:40.686084+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:40.686084+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11023986,"num_objects":1346,"num_object_clones":0,"num_object_copies":5384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10737,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10622","reported_seq":10715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.667467+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:40.667467+0000","last_peered":"2024-09-15T03:15:40.667467+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:40.667467+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11037924,"num_objects":1348,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10619,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10560","reported_seq":10681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:35.898024+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:35.898024+0000","last_peered":"2024-09-15T03:15:35.898024+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:35.898024+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11531890,"num_objects":1408,"num_object_clones":0,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10597","reported_seq":10692,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.776023+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:40.776023+0000","last_peered":"2024-09-15T03:15:40.776023+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:40.776023+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10531243,"num_objects":1286,"num_object_clones":0,"num_object_copies":5144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10555","reported_seq":14597,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:38.687382+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:38.687382+0000","last_peered":"2024-09-15T03:15:38.687382+0000","last_clean":"2024-09-15T03:15:38.687382+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:38.687382+0000","last_undegraded":"2024-09-15T03:15:38.687382+0000","last_fullsized":"2024-09-15T03:15:38.687382+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11008825,"num_objects":1344,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10719","reported_seq":14895,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:38.654520+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:38.654520+0000","last_peered":"2024-09-15T03:15:38.654520+0000","last_clean":"2024-09-15T03:15:38.654520+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:38.654520+0000","last_undegraded":"2024-09-15T03:15:38.654520+0000","last_fullsized":"2024-09-15T03:15:38.654520+0000","mapping_epoch":36,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11582265,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10392","reported_seq":10493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.682256+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:40.682256+0000","last_peered":"2024-09-15T03:15:40.682256+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:40.682256+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10565234,"num_objects":1290,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-723}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10503","reported_seq":10612,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:35.910509+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:35.910509+0000","last_peered":"2024-09-15T03:15:35.910509+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:35.910509+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11302514,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10501,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-597}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10681","reported_seq":10796,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.468924+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:40.468924+0000","last_peered":"2024-09-15T03:15:40.468924+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:40.468924+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10876530,"num_objects":1328,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10679,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":175741524,"num_objects":21458,"num_object_clones":0,"num_object_copies":85832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21458,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169249,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6612,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":352608256,"data_stored":352608256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160876,"ondisk_log_size":160876,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675732,"kb_used_data":85324,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93696108,"statfs":{"total":96636764160,"available":95944814592,"internally_reserved":0,"allocated":87371776,"data_stored":87187955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":8.4090000000000007}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":2.5089999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.45200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":12.426},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.34399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678920,"kb_used_data":89024,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93692920,"statfs":{"total":96636764160,"available":95941550080,"internally_reserved":0,"allocated":91160576,"data_stored":90979901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,6,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":9.2420000000000009},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.20200000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.4380000000000002},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":3.464}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":3.2559999999999998},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":0.372}]}]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397208,"kb_used_data":86864,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93974632,"statfs":{"total":96636764160,"available":96230023168,"internally_reserved":0,"allocated":88948736,"data_stored":88764915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,6,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.19400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.215},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":0.27000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.44},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.53700000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684004,"kb_used_data":85340,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93687836,"statfs":{"total":96636764160,"available":95936344064,"internally_reserved":0,"allocated":87388160,"data_stored":87204339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":7.6790000000000003},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.74199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.251},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":11.657}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":11.708},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":11.734}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86855680,"data_stored":86855680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88518656,"data_stored":88518656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90406912,"data_stored":90406912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86827008,"data_stored":86827008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:41.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:43.309 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:43.326 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:43.396 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-15T03:15:42.929385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174291540,"num_objects":21281,"num_object_clones":0,"num_object_copies":85124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169426,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6612,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2436472,"kb_used_data":343000,"kb_used_omap":14,"kb_used_meta":2093425,"kb_avail":375050888,"statfs":{"total":386547056640,"available":384052109312,"internally_reserved":0,"allocated":351232000,"data_stored":350499862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2143667227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,6,9,12,9,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15958016,"num_objects":-1948,"num_object_clones":0,"num_object_copies":-7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1948,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"1.f","version":"37'10589","reported_seq":10657,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.768810+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:40.768810+0000","last_peered":"2024-09-15T03:15:40.768810+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:40.768810+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10821632,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10181","reported_seq":11030,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.767404+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:40.767404+0000","last_peered":"2024-09-15T03:15:40.767404+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:40.767404+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10196594,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10179,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10841","reported_seq":10951,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.934509+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:40.934509+0000","last_peered":"2024-09-15T03:15:40.934509+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:40.934509+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11295569,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10840,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10305","reported_seq":14279,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:38.771428+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:38.771428+0000","last_peered":"2024-09-15T03:15:38.771428+0000","last_clean":"2024-09-15T03:15:38.771428+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:38.771428+0000","last_undegraded":"2024-09-15T03:15:38.771428+0000","last_fullsized":"2024-09-15T03:15:38.771428+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10835570,"num_objects":1323,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10303,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10718","reported_seq":10826,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.715550+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:40.715550+0000","last_peered":"2024-09-15T03:15:40.715550+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:40.715550+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11163250,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10716,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10692","reported_seq":10778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.659293+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:40.659293+0000","last_peered":"2024-09-15T03:15:40.659293+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:40.659293+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10698752,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10642","reported_seq":10764,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.836067+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:40.836067+0000","last_peered":"2024-09-15T03:15:40.836067+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:40.836067+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10778226,"num_objects":1316,"num_object_clones":0,"num_object_copies":5264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10640,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-710}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10739","reported_seq":10858,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.686084+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:40.686084+0000","last_peered":"2024-09-15T03:15:40.686084+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:40.686084+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11023986,"num_objects":1346,"num_object_clones":0,"num_object_copies":5384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10737,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10622","reported_seq":10715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.667467+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:40.667467+0000","last_peered":"2024-09-15T03:15:40.667467+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:40.667467+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11037924,"num_objects":1348,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10619,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10615","reported_seq":10736,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:41.042383+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:41.042383+0000","last_peered":"2024-09-15T03:15:41.042383+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:41.042383+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11081330,"num_objects":1353,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10597","reported_seq":10692,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.776023+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:40.776023+0000","last_peered":"2024-09-15T03:15:40.776023+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:40.776023+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10531243,"num_objects":1286,"num_object_clones":0,"num_object_copies":5144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10555","reported_seq":14597,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:38.687382+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:38.687382+0000","last_peered":"2024-09-15T03:15:38.687382+0000","last_clean":"2024-09-15T03:15:38.687382+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:38.687382+0000","last_undegraded":"2024-09-15T03:15:38.687382+0000","last_fullsized":"2024-09-15T03:15:38.687382+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11008825,"num_objects":1344,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10719","reported_seq":14895,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:38.654520+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:38.654520+0000","last_peered":"2024-09-15T03:15:38.654520+0000","last_clean":"2024-09-15T03:15:38.654520+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:38.654520+0000","last_undegraded":"2024-09-15T03:15:38.654520+0000","last_fullsized":"2024-09-15T03:15:38.654520+0000","mapping_epoch":36,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11582265,"num_objects":1414,"num_object_clones":0,"num_object_copies":5656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10392","reported_seq":10493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.682256+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:40.682256+0000","last_peered":"2024-09-15T03:15:40.682256+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:40.682256+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10565234,"num_objects":1290,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-723}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10565","reported_seq":10674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.904536+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:40.904536+0000","last_peered":"2024-09-15T03:15:40.904536+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:40.904536+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10794610,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10563,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-659}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10681","reported_seq":10796,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.468924+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:40.468924+0000","last_peered":"2024-09-15T03:15:40.468924+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:40.468924+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10876530,"num_objects":1328,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10679,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":174291540,"num_objects":21281,"num_object_clones":0,"num_object_copies":85124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169426,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6612,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":348868608,"data_stored":348868608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160853,"ondisk_log_size":160853,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675732,"kb_used_data":85324,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93696108,"statfs":{"total":96636764160,"available":95944814592,"internally_reserved":0,"allocated":87371776,"data_stored":87187955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":8.4090000000000007}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":2.5089999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.45200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":12.426},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.34399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679528,"kb_used_data":85472,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93692312,"statfs":{"total":96636764160,"available":95940927488,"internally_reserved":0,"allocated":87523328,"data_stored":87342653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":2.6920000000000002},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.25900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":8.7530000000000001},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":8.7940000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":17.981000000000002},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":22.367999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":397208,"kb_used_data":86864,"kb_used_omap":4,"kb_used_meta":310331,"kb_avail":93974632,"statfs":{"total":96636764160,"available":96230023168,"internally_reserved":0,"allocated":88948736,"data_stored":88764915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":317779521},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,6,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.19400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.215},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":0.27000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.44},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.53700000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684004,"kb_used_data":85340,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93687836,"statfs":{"total":96636764160,"available":95936344064,"internally_reserved":0,"allocated":87388160,"data_stored":87204339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":7.6790000000000003},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.74199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.251},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":11.657}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":11.708},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":11.734}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86855680,"data_stored":86855680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88518656,"data_stored":88518656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86667264,"data_stored":86667264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86827008,"data_stored":86827008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:43.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:15:43.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10589", 'reported_seq': 10657, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.768810+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:15:40.768810+0000', 'last_peered': '2024-09-15T03:15:40.768810+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:15:40.768810+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10821632, 'num_objects': 1321, 'num_object_clones': 0, 'num_object_copies': 5284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10589, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -645}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:43.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10181", 'reported_seq': 11030, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.767404+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:15:40.767404+0000', 'last_peered': '2024-09-15T03:15:40.767404+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:15:40.767404+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10196594, 'num_objects': 1245, 'num_object_clones': 0, 'num_object_copies': 4980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1610, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10179, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 392, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -365}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10841", 'reported_seq': 10951, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.934509+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:15:40.934509+0000', 'last_peered': '2024-09-15T03:15:40.934509+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:15:40.934509+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11295569, 'num_objects': 1380, 'num_object_clones': 0, 'num_object_copies': 5520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1380, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10840, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -711}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10718", 'reported_seq': 10826, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.715550+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:15:40.715550+0000', 'last_peered': '2024-09-15T03:15:40.715550+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:15:40.715550+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11163250, 'num_objects': 1363, 'num_object_clones': 0, 'num_object_copies': 5452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10716, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -721}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10692", 'reported_seq': 10778, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.659293+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:15:40.659293+0000', 'last_peered': '2024-09-15T03:15:40.659293+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:15:40.659293+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10698752, 'num_objects': 1306, 'num_object_clones': 0, 'num_object_copies': 5224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10692, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -732}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10642", 'reported_seq': 10764, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.836067+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:15:40.836067+0000', 'last_peered': '2024-09-15T03:15:40.836067+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:15:40.836067+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10778226, 'num_objects': 1316, 'num_object_clones': 0, 'num_object_copies': 5264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10640, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -710}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:15:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10739", 'reported_seq': 10858, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.686084+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:15:40.686084+0000', 'last_peered': '2024-09-15T03:15:40.686084+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:15:40.686084+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11023986, 'num_objects': 1346, 'num_object_clones': 0, 'num_object_copies': 5384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10737, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10622", 'reported_seq': 10715, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.667467+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:15:40.667467+0000', 'last_peered': '2024-09-15T03:15:40.667467+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:15:40.667467+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11037924, 'num_objects': 1348, 'num_object_clones': 0, 'num_object_copies': 5392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10619, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -679}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10615", 'reported_seq': 10736, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:41.042383+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:15:41.042383+0000', 'last_peered': '2024-09-15T03:15:41.042383+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:15:41.042383+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11081330, 'num_objects': 1353, 'num_object_clones': 0, 'num_object_copies': 5412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10613, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -689}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10597", 'reported_seq': 10692, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.776023+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:15:40.776023+0000', 'last_peered': '2024-09-15T03:15:40.776023+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:15:40.776023+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10531243, 'num_objects': 1286, 'num_object_clones': 0, 'num_object_copies': 5144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10594, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -776}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10392", 'reported_seq': 10493, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.682256+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:15:40.682256+0000', 'last_peered': '2024-09-15T03:15:40.682256+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:15:40.682256+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10565234, 'num_objects': 1290, 'num_object_clones': 0, 'num_object_copies': 5160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10390, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -723}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:15:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10565", 'reported_seq': 10674, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.904536+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:15:40.904536+0000', 'last_peered': '2024-09-15T03:15:40.904536+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:15:40.904536+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10794610, 'num_objects': 1318, 'num_object_clones': 0, 'num_object_copies': 5272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10563, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -659}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:15:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10681", 'reported_seq': 10796, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:40.468924+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:15:40.468924+0000', 'last_peered': '2024-09-15T03:15:40.468924+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:15:40.468924+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10876530, 'num_objects': 1328, 'num_object_clones': 0, 'num_object_copies': 5312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10679, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -731}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:43.400 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:15:44.916 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:44.916 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26072/85124 objects degraded (30.628%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":667,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21281,"data_bytes":174291540,"bytes_used":2494947328,"bytes_avail":384052109312,"bytes_total":386547056640,"degraded_objects":26072,"degraded_total":85124,"degraded_ratio":0.30628259950190312,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":162},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:15:44.967 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:15:44.967 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:46.918 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:46.942 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:46.961 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-15T03:15:44.929851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":172898900,"num_objects":21111,"num_object_clones":0,"num_object_copies":84444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169596,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6612,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2420344,"kb_used_data":339352,"kb_used_omap":14,"kb_used_meta":2080945,"kb_avail":375067016,"statfs":{"total":386547056640,"available":384068624384,"internally_reserved":0,"allocated":347496448,"data_stored":346764310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130887707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,8,7,9,9,8,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16039936,"num_objects":-1958,"num_object_clones":0,"num_object_copies":-7832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1958,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002946"},"pg_stats":[{"pgid":"1.f","version":"37'10589","reported_seq":10657,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.768810+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:40.768810+0000","last_peered":"2024-09-15T03:15:40.768810+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:40.768810+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10821632,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10589,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10181","reported_seq":11030,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.767404+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:40.767404+0000","last_peered":"2024-09-15T03:15:40.767404+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:40.767404+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10196594,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10179,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":392,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10841","reported_seq":10951,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.934509+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:40.934509+0000","last_peered":"2024-09-15T03:15:40.934509+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:40.934509+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11295569,"num_objects":1380,"num_object_clones":0,"num_object_copies":5520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10840,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10365","reported_seq":14337,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:43.811635+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:43.811635+0000","last_peered":"2024-09-15T03:15:43.811635+0000","last_clean":"2024-09-15T03:15:43.811635+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:43.811635+0000","last_undegraded":"2024-09-15T03:15:43.811635+0000","last_fullsized":"2024-09-15T03:15:43.811635+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10344050,"num_objects":1263,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10363,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10718","reported_seq":10826,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.715550+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:40.715550+0000","last_peered":"2024-09-15T03:15:40.715550+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:40.715550+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11163250,"num_objects":1363,"num_object_clones":0,"num_object_copies":5452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10716,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10692","reported_seq":10778,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.659293+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:40.659293+0000","last_peered":"2024-09-15T03:15:40.659293+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:40.659293+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10698752,"num_objects":1306,"num_object_clones":0,"num_object_copies":5224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10642","reported_seq":10764,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.836067+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:40.836067+0000","last_peered":"2024-09-15T03:15:40.836067+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:40.836067+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10778226,"num_objects":1316,"num_object_clones":0,"num_object_copies":5264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10640,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-710}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10739","reported_seq":10858,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.686084+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:40.686084+0000","last_peered":"2024-09-15T03:15:40.686084+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:40.686084+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11023986,"num_objects":1346,"num_object_clones":0,"num_object_copies":5384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10737,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10622","reported_seq":10715,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.667467+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:40.667467+0000","last_peered":"2024-09-15T03:15:40.667467+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:40.667467+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11037924,"num_objects":1348,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10619,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10615","reported_seq":10736,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:41.042383+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:41.042383+0000","last_peered":"2024-09-15T03:15:41.042383+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:41.042383+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11081330,"num_objects":1353,"num_object_clones":0,"num_object_copies":5412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10597","reported_seq":10692,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.776023+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:40.776023+0000","last_peered":"2024-09-15T03:15:40.776023+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:40.776023+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10531243,"num_objects":1286,"num_object_clones":0,"num_object_copies":5144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10607","reported_seq":14650,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:43.860549+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:43.860549+0000","last_peered":"2024-09-15T03:15:43.860549+0000","last_clean":"2024-09-15T03:15:43.860549+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:43.860549+0000","last_undegraded":"2024-09-15T03:15:43.860549+0000","last_fullsized":"2024-09-15T03:15:43.860549+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10582841,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10777","reported_seq":14953,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:43.818334+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:43.818334+0000","last_peered":"2024-09-15T03:15:43.818334+0000","last_clean":"2024-09-15T03:15:43.818334+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:43.818334+0000","last_undegraded":"2024-09-15T03:15:43.818334+0000","last_fullsized":"2024-09-15T03:15:43.818334+0000","mapping_epoch":36,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11107129,"num_objects":1356,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10392","reported_seq":10493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.682256+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:40.682256+0000","last_peered":"2024-09-15T03:15:40.682256+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:40.682256+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10565234,"num_objects":1290,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10390,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-723}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10565","reported_seq":10674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.904536+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:40.904536+0000","last_peered":"2024-09-15T03:15:40.904536+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:40.904536+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10794610,"num_objects":1318,"num_object_clones":0,"num_object_copies":5272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10563,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-659}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10681","reported_seq":10796,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:40.468924+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:40.468924+0000","last_peered":"2024-09-15T03:15:40.468924+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:40.468924+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10876530,"num_objects":1328,"num_object_clones":0,"num_object_copies":5312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10679,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":172898900,"num_objects":21111,"num_object_clones":0,"num_object_copies":84444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169596,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6612,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":345161728,"data_stored":345161728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160923,"ondisk_log_size":160923,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675732,"kb_used_data":85324,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93696108,"statfs":{"total":96636764160,"available":95944814592,"internally_reserved":0,"allocated":87371776,"data_stored":87187955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":8.4090000000000007}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":2.5089999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.45200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":12.426},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.34399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679528,"kb_used_data":85472,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93692312,"statfs":{"total":96636764160,"available":95940927488,"internally_reserved":0,"allocated":87523328,"data_stored":87342653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.718,"15min":1.7370000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":12.497999999999999,"5min":31.175000000000001,"15min":31.175000000000001},"last":2.6920000000000002},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.8280000000000001,"15min":1.349},"min":{"1min":0.20000000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.573,"5min":19.329000000000001,"15min":19.329000000000001},"last":0.25900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":8.7530000000000001},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":8.7940000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:14:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1640000000000001,"5min":2.1459999999999999,"15min":1.4810000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.216},"max":{"1min":11.712,"5min":14.709,"15min":21.600999999999999},"last":17.981000000000002},{"interface":"front","average":{"1min":1.6850000000000001,"5min":2.4329999999999998,"15min":1.585},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":8.7349999999999994,"5min":22.710000000000001,"15min":22.710000000000001},"last":22.367999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381080,"kb_used_data":83216,"kb_used_omap":4,"kb_used_meta":297851,"kb_avail":93990760,"statfs":{"total":96636764160,"available":96246538240,"internally_reserved":0,"allocated":85213184,"data_stored":85029363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":305000001},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":1.7330000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":2.3799999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":8.9860000000000007},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.33000000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":684004,"kb_used_data":85340,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93687836,"statfs":{"total":96636764160,"available":95936344064,"internally_reserved":0,"allocated":87388160,"data_stored":87204339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":7.6790000000000003},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.74199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.251},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":11.657}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":11.708},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":11.734}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86855680,"data_stored":86855680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":84811776,"data_stored":84811776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86667264,"data_stored":86667264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86827008,"data_stored":86827008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:49.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:51.896 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:51.900 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:51.923 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-15T03:15:50.935725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165632596,"num_objects":20224,"num_object_clones":0,"num_object_copies":80896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170483,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6653,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2406348,"kb_used_data":325356,"kb_used_omap":14,"kb_used_meta":2080945,"kb_avail":375081012,"statfs":{"total":386547056640,"available":384082956288,"internally_reserved":0,"allocated":333164544,"data_stored":332432406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130887707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,3,13,15,17,15,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14819328,"num_objects":-1809,"num_object_clones":0,"num_object_copies":-7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1809,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007318"},"pg_stats":[{"pgid":"1.f","version":"37'10644","reported_seq":10711,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.811839+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:45.811839+0000","last_peered":"2024-09-15T03:15:45.811839+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:45.811839+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10371072,"num_objects":1266,"num_object_clones":0,"num_object_copies":5064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10233","reported_seq":11164,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.781741+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:45.781741+0000","last_peered":"2024-09-15T03:15:45.781741+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:45.781741+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9770610,"num_objects":1193,"num_object_clones":0,"num_object_copies":4772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10901","reported_seq":11011,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:46.047486+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:46.047486+0000","last_peered":"2024-09-15T03:15:46.047486+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:46.047486+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10804049,"num_objects":1320,"num_object_clones":0,"num_object_copies":5280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10900,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-771}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10417","reported_seq":14391,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:48.967793+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:48.967793+0000","last_peered":"2024-09-15T03:15:48.967793+0000","last_clean":"2024-09-15T03:15:48.967793+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:48.967793+0000","last_undegraded":"2024-09-15T03:15:48.967793+0000","last_fullsized":"2024-09-15T03:15:48.967793+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9918066,"num_objects":1211,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10415,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10779","reported_seq":10887,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.753838+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:45.753838+0000","last_peered":"2024-09-15T03:15:45.753838+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:45.753838+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10663538,"num_objects":1302,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10745","reported_seq":10831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.665876+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:45.665876+0000","last_peered":"2024-09-15T03:15:45.665876+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:45.665876+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10264576,"num_objects":1253,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10745,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10689","reported_seq":10812,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.619163+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:45.619163+0000","last_peered":"2024-09-15T03:15:45.619163+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:45.619163+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10393202,"num_objects":1269,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10793","reported_seq":10912,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.818492+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:45.818492+0000","last_peered":"2024-09-15T03:15:45.818492+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:45.818492+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10581618,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10680","reported_seq":10773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.831998+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:45.831998+0000","last_peered":"2024-09-15T03:15:45.831998+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:45.831998+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10562788,"num_objects":1290,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10677,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10671","reported_seq":10792,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.890094+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:45.890094+0000","last_peered":"2024-09-15T03:15:45.890094+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:45.890094+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10622578,"num_objects":1297,"num_object_clones":0,"num_object_copies":5188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10669,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-745}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10662","reported_seq":10755,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.754867+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:45.754867+0000","last_peered":"2024-09-15T03:15:45.754867+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:45.754867+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9998763,"num_objects":1221,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10659,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-841}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10658","reported_seq":14701,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:49.058095+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:49.058095+0000","last_peered":"2024-09-15T03:15:49.058095+0000","last_clean":"2024-09-15T03:15:49.058095+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:49.058095+0000","last_undegraded":"2024-09-15T03:15:49.058095+0000","last_fullsized":"2024-09-15T03:15:49.058095+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10165049,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10845","reported_seq":15021,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:48.974591+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:48.974591+0000","last_peered":"2024-09-15T03:15:48.974591+0000","last_clean":"2024-09-15T03:15:48.974591+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:48.974591+0000","last_undegraded":"2024-09-15T03:15:48.974591+0000","last_fullsized":"2024-09-15T03:15:48.974591+0000","mapping_epoch":36,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10550073,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10437","reported_seq":10539,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.568892+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:45.568892+0000","last_peered":"2024-09-15T03:15:45.568892+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:45.568892+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10196594,"num_objects":1245,"num_object_clones":0,"num_object_copies":4980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10435,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-768}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10612","reported_seq":10721,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.939030+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:45.939030+0000","last_peered":"2024-09-15T03:15:45.939030+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:45.939030+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10409586,"num_objects":1271,"num_object_clones":0,"num_object_copies":5084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10610,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-706}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10744","reported_seq":10859,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:45.535467+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:45.535467+0000","last_peered":"2024-09-15T03:15:45.535467+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:45.535467+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10360434,"num_objects":1265,"num_object_clones":0,"num_object_copies":5060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165632596,"num_objects":20224,"num_object_clones":0,"num_object_copies":80896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170483,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6653,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":330670080,"data_stored":330670080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676308,"kb_used_data":81740,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93695532,"statfs":{"total":96636764160,"available":95944224768,"internally_reserved":0,"allocated":83701760,"data_stored":83517939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.3759999999999999,"15min":1.7050000000000001},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":20.984000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.21299999999999999},{"interface":"front","average":{"1min":2.2400000000000002,"5min":1.845,"15min":2.0459999999999998},"min":{"1min":0.23699999999999999,"5min":0.185,"15min":0.185},"max":{"1min":9.6370000000000005,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.247}]},{"osd":2,"last update":"Sun Sep 15 03:14:46 2024","interfaces":[{"interface":"back","average":{"1min":4.25,"5min":3.903,"15min":2.0739999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":24.82,"5min":40.784999999999997,"15min":40.784999999999997},"last":3.169},{"interface":"front","average":{"1min":3.2269999999999999,"5min":2.226,"15min":1.6619999999999999},"min":{"1min":0.23200000000000001,"5min":0.22600000000000001,"15min":0.188},"max":{"1min":21.062000000000001,"5min":32.377000000000002,"15min":32.377000000000002},"last":2.8039999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":680240,"kb_used_data":82024,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93691600,"statfs":{"total":96636764160,"available":95940198400,"internally_reserved":0,"allocated":83992576,"data_stored":83811901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,5,6,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.29399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.42399999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.36099999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381764,"kb_used_data":79740,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93990076,"statfs":{"total":96636764160,"available":96245837824,"internally_reserved":0,"allocated":81653760,"data_stored":81469939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,6,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.313},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":3.8820000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.442},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.33500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668036,"kb_used_data":81852,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93703804,"statfs":{"total":96636764160,"available":95952695296,"internally_reserved":0,"allocated":83816448,"data_stored":83632627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":7.6790000000000003},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.74199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4340000000000002,"5min":2.8140000000000001,"15min":1.946},"min":{"1min":0.193,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":7.9290000000000003,"5min":30.306000000000001,"15min":30.802},"last":0.251},{"interface":"front","average":{"1min":1.5089999999999999,"5min":2.3130000000000002,"15min":1.361},"min":{"1min":0.25900000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.042,"5min":20.387,"15min":20.387},"last":11.657}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":11.708},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":11.734}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":83247104,"data_stored":83247104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81113088,"data_stored":81113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":83091456,"data_stored":83091456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83218432,"data_stored":83218432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:51.924 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:53.934 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:53.944 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:53.958 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-15T03:15:52.936214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":159693396,"num_objects":19499,"num_object_clones":0,"num_object_copies":77996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171208,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6710,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2391584,"kb_used_data":314752,"kb_used_omap":14,"kb_used_meta":2076785,"kb_avail":375095776,"statfs":{"total":386547056640,"available":384098074624,"internally_reserved":0,"allocated":322306048,"data_stored":321573910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126627867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,11,12,14,18,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16048128,"num_objects":-1959,"num_object_clones":0,"num_object_copies":-7836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1959,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007328"},"pg_stats":[{"pgid":"1.f","version":"37'10696","reported_seq":10764,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.792536+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:50.792536+0000","last_peered":"2024-09-15T03:15:50.792536+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:50.792536+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9945088,"num_objects":1214,"num_object_clones":0,"num_object_copies":4856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10696,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-752}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10283","reported_seq":11328,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:51.020521+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:51.020521+0000","last_peered":"2024-09-15T03:15:51.020521+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:51.020521+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9361010,"num_objects":1143,"num_object_clones":0,"num_object_copies":4572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10281,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":490,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'10961","reported_seq":11069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:51.002753+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:51.002753+0000","last_peered":"2024-09-15T03:15:51.002753+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:51.002753+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10312529,"num_objects":1260,"num_object_clones":0,"num_object_copies":5040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1260,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10960,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-831}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10417","reported_seq":14391,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:48.967793+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:48.967793+0000","last_peered":"2024-09-15T03:15:48.967793+0000","last_clean":"2024-09-15T03:15:48.967793+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:48.967793+0000","last_undegraded":"2024-09-15T03:15:48.967793+0000","last_fullsized":"2024-09-15T03:15:48.967793+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9918066,"num_objects":1211,"num_object_clones":0,"num_object_copies":4844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10415,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10829","reported_seq":10937,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.829658+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:50.829658+0000","last_peered":"2024-09-15T03:15:50.829658+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:50.829658+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10253938,"num_objects":1252,"num_object_clones":0,"num_object_copies":5008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-832}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10797","reported_seq":10883,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.769765+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:50.769765+0000","last_peered":"2024-09-15T03:15:50.769765+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:50.769765+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9838592,"num_objects":1201,"num_object_clones":0,"num_object_copies":4804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10797,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10749","reported_seq":10872,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.940571+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:50.940571+0000","last_peered":"2024-09-15T03:15:50.940571+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:50.940571+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9901682,"num_objects":1209,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10747,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-817}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10856","reported_seq":10975,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.903961+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:50.903961+0000","last_peered":"2024-09-15T03:15:50.903961+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:50.903961+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065522,"num_objects":1229,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10734","reported_seq":10827,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.484717+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:50.484717+0000","last_peered":"2024-09-15T03:15:50.484717+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:50.484717+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10120420,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10731,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-791}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10738","reported_seq":10856,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.976733+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:50.976733+0000","last_peered":"2024-09-15T03:15:50.976733+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:50.976733+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10073714,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10736,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-812}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10709","reported_seq":10805,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.631975+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:50.631975+0000","last_peered":"2024-09-15T03:15:50.631975+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:50.631975+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9613739,"num_objects":1174,"num_object_clones":0,"num_object_copies":4696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-888}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10658","reported_seq":14701,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:49.058095+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:49.058095+0000","last_peered":"2024-09-15T03:15:49.058095+0000","last_clean":"2024-09-15T03:15:49.058095+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:49.058095+0000","last_undegraded":"2024-09-15T03:15:49.058095+0000","last_fullsized":"2024-09-15T03:15:49.058095+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10165049,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10657,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10845","reported_seq":15021,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:48.974591+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:48.974591+0000","last_peered":"2024-09-15T03:15:48.974591+0000","last_clean":"2024-09-15T03:15:48.974591+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:48.974591+0000","last_undegraded":"2024-09-15T03:15:48.974591+0000","last_fullsized":"2024-09-15T03:15:48.974591+0000","mapping_epoch":36,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10550073,"num_objects":1288,"num_object_clones":0,"num_object_copies":5152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10505","reported_seq":10607,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.912999+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:50.912999+0000","last_peered":"2024-09-15T03:15:50.912999+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:50.912999+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9639538,"num_objects":1177,"num_object_clones":0,"num_object_copies":4708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-836}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10664","reported_seq":10772,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:51.008622+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:51.008622+0000","last_peered":"2024-09-15T03:15:51.008622+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:51.008622+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983602,"num_objects":1219,"num_object_clones":0,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10662,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-758}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10794","reported_seq":10909,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:50.768532+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:50.768532+0000","last_peered":"2024-09-15T03:15:50.768532+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:50.768532+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9950834,"num_objects":1215,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":159693396,"num_objects":19499,"num_object_clones":0,"num_object_copies":77996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171208,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6710,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":319709184,"data_stored":319709184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160835,"ondisk_log_size":160835,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676920,"kb_used_data":78192,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93694920,"statfs":{"total":96636764160,"available":95943598080,"internally_reserved":0,"allocated":80068608,"data_stored":79884787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.20200000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":23.812000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":5.625}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":4.0720000000000001},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.32000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664288,"kb_used_data":78552,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93707552,"statfs":{"total":96636764160,"available":95956533248,"internally_reserved":0,"allocated":80437248,"data_stored":80256573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.214},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.29199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.501},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.32700000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381764,"kb_used_data":79740,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":93990076,"statfs":{"total":96636764160,"available":96245837824,"internally_reserved":0,"allocated":81653760,"data_stored":81469939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,5,6,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.313},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.27700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":3.8820000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.442},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.33500000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":668612,"kb_used_data":78268,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93703228,"statfs":{"total":96636764160,"available":95952105472,"internally_reserved":0,"allocated":80146432,"data_stored":79962611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,5,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.878},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.81499999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.84399999999999997},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.58799999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.53,"15min":1.284},"min":{"1min":0.26300000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":9.0860000000000003,"5min":20.199000000000002,"15min":20.948},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.2389999999999999,"5min":2.2970000000000002,"15min":1.3680000000000001},"min":{"1min":0.29299999999999998,"5min":0.26100000000000001,"15min":0.22700000000000001},"max":{"1min":16.562000000000001,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.43099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79560704,"data_stored":79560704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81113088,"data_stored":81113088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":79441920,"data_stored":79441920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":79593472,"data_stored":79593472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:15:53.960 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:15:53.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10696", 'reported_seq': 10764, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.792536+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:15:50.792536+0000', 'last_peered': '2024-09-15T03:15:50.792536+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:15:50.792536+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9945088, 'num_objects': 1214, 'num_object_clones': 0, 'num_object_copies': 4856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10696, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -752}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:53.960 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:15:53.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10283", 'reported_seq': 11328, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:51.020521+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:15:51.020521+0000', 'last_peered': '2024-09-15T03:15:51.020521+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:15:51.020521+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9361010, 'num_objects': 1143, 'num_object_clones': 0, 'num_object_copies': 4572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1512, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1143, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10281, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 490, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -369}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'10961", 'reported_seq': 11069, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:51.002753+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:15:51.002753+0000', 'last_peered': '2024-09-15T03:15:51.002753+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:15:51.002753+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10312529, 'num_objects': 1260, 'num_object_clones': 0, 'num_object_copies': 5040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1260, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10960, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -831}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10829", 'reported_seq': 10937, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.829658+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:15:50.829658+0000', 'last_peered': '2024-09-15T03:15:50.829658+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:15:50.829658+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10253938, 'num_objects': 1252, 'num_object_clones': 0, 'num_object_copies': 5008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10827, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -832}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10797", 'reported_seq': 10883, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.769765+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:15:50.769765+0000', 'last_peered': '2024-09-15T03:15:50.769765+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:15:50.769765+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9838592, 'num_objects': 1201, 'num_object_clones': 0, 'num_object_copies': 4804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10797, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -837}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10749", 'reported_seq': 10872, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.940571+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:15:50.940571+0000', 'last_peered': '2024-09-15T03:15:50.940571+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:15:50.940571+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9901682, 'num_objects': 1209, 'num_object_clones': 0, 'num_object_copies': 4836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10747, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -817}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10856", 'reported_seq': 10975, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.903961+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:15:50.903961+0000', 'last_peered': '2024-09-15T03:15:50.903961+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:15:50.903961+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10065522, 'num_objects': 1229, 'num_object_clones': 0, 'num_object_copies': 4916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10854, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -827}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:53.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10734", 'reported_seq': 10827, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.484717+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:15:50.484717+0000', 'last_peered': '2024-09-15T03:15:50.484717+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:15:50.484717+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10120420, 'num_objects': 1236, 'num_object_clones': 0, 'num_object_copies': 4944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10731, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -791}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10738", 'reported_seq': 10856, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.976733+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:15:50.976733+0000', 'last_peered': '2024-09-15T03:15:50.976733+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:15:50.976733+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10073714, 'num_objects': 1230, 'num_object_clones': 0, 'num_object_copies': 4920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10736, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -812}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10709", 'reported_seq': 10805, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.631975+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:15:50.631975+0000', 'last_peered': '2024-09-15T03:15:50.631975+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:15:50.631975+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9613739, 'num_objects': 1174, 'num_object_clones': 0, 'num_object_copies': 4696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1174, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10706, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -888}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10505", 'reported_seq': 10607, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.912999+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:15:50.912999+0000', 'last_peered': '2024-09-15T03:15:50.912999+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:15:50.912999+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9639538, 'num_objects': 1177, 'num_object_clones': 0, 'num_object_copies': 4708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10503, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -836}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10664", 'reported_seq': 10772, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:51.008622+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:15:51.008622+0000', 'last_peered': '2024-09-15T03:15:51.008622+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:15:51.008622+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9983602, 'num_objects': 1219, 'num_object_clones': 0, 'num_object_copies': 4876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10662, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -758}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:15:53.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10794", 'reported_seq': 10909, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:15:50.768532+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:15:50.768532+0000', 'last_peered': '2024-09-15T03:15:50.768532+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:15:50.768532+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9950834, 'num_objects': 1215, 'num_object_clones': 0, 'num_object_copies': 4860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10792, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:15:53.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:15:56.111 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:56.111 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25974/77340 objects degraded (33.584%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":679,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":19335,"data_bytes":158349908,"bytes_used":2449530880,"bytes_avail":384097525760,"bytes_total":386547056640,"degraded_objects":25974,"degraded_total":77340,"degraded_ratio":0.33584173778122578,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":162},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:15:56.130 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:15:56.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:15:57.851 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:15:57.897 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:15:57.934 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-15T03:15:56.940417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":152500820,"num_objects":18621,"num_object_clones":0,"num_object_copies":74484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172086,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6760,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160713,"ondisk_log_size":160713,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2377552,"kb_used_data":300696,"kb_used_omap":14,"kb_used_meta":2076785,"kb_avail":375109808,"statfs":{"total":386547056640,"available":384112443392,"internally_reserved":0,"allocated":307912704,"data_stored":307180566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2126627867},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,1,7,5,12,6,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20398080,"num_objects":-2490,"num_object_clones":0,"num_object_copies":-9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010565"},"pg_stats":[{"pgid":"1.f","version":"37'10760","reported_seq":10828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.886368+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:55.886368+0000","last_peered":"2024-09-15T03:15:55.886368+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:55.886368+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9420800,"num_objects":1150,"num_object_clones":0,"num_object_copies":4600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10335","reported_seq":11480,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:56.029789+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:56.029789+0000","last_peered":"2024-09-15T03:15:56.029789+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:56.029789+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8935026,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":540,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11022","reported_seq":11130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.919759+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:55.919759+0000","last_peered":"2024-09-15T03:15:55.919759+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:55.919759+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9812817,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11021,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10463","reported_seq":14437,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:54.212624+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:54.212624+0000","last_peered":"2024-09-15T03:15:54.212624+0000","last_clean":"2024-09-15T03:15:54.212624+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:54.212624+0000","last_undegraded":"2024-09-15T03:15:54.212624+0000","last_fullsized":"2024-09-15T03:15:54.212624+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9541234,"num_objects":1165,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10896","reported_seq":11004,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.689935+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:55.689935+0000","last_peered":"2024-09-15T03:15:55.689935+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:55.689935+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9705074,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10845","reported_seq":10931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.855696+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:55.855696+0000","last_peered":"2024-09-15T03:15:55.855696+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:55.855696+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9445376,"num_objects":1153,"num_object_clones":0,"num_object_copies":4612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10845,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10812","reported_seq":10935,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.969790+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:55.969790+0000","last_peered":"2024-09-15T03:15:55.969790+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:55.969790+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9385586,"num_objects":1146,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10919","reported_seq":11038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:56.038699+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:56.038699+0000","last_peered":"2024-09-15T03:15:56.038699+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:56.038699+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9549426,"num_objects":1166,"num_object_clones":0,"num_object_copies":4664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10917,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10782","reported_seq":10875,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.693967+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:55.693967+0000","last_peered":"2024-09-15T03:15:55.693967+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:55.693967+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9727204,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10789","reported_seq":10910,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:56.075659+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:56.075659+0000","last_peered":"2024-09-15T03:15:56.075659+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:56.075659+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9655922,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-863}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10758","reported_seq":10854,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.744883+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:55.744883+0000","last_peered":"2024-09-15T03:15:55.744883+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:55.744883+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9212331,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10709","reported_seq":14752,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:54.157581+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:54.157581+0000","last_peered":"2024-09-15T03:15:54.157581+0000","last_clean":"2024-09-15T03:15:54.157581+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:54.157581+0000","last_undegraded":"2024-09-15T03:15:54.157581+0000","last_fullsized":"2024-09-15T03:15:54.157581+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9747257,"num_objects":1190,"num_object_clones":0,"num_object_copies":4760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10708,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10912","reported_seq":15085,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:54.212840+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:54.212840+0000","last_peered":"2024-09-15T03:15:54.212840+0000","last_clean":"2024-09-15T03:15:54.212840+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:54.212840+0000","last_undegraded":"2024-09-15T03:15:54.212840+0000","last_fullsized":"2024-09-15T03:15:54.212840+0000","mapping_epoch":36,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10001209,"num_objects":1221,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10911,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10557","reported_seq":10659,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.898895+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:55.898895+0000","last_peered":"2024-09-15T03:15:55.898895+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:55.898895+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9213554,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10555,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10704","reported_seq":10813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.956913+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:55.956913+0000","last_peered":"2024-09-15T03:15:55.956913+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:55.956913+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9655922,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10850","reported_seq":10965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.953729+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:55.953729+0000","last_peered":"2024-09-15T03:15:55.953729+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:55.953729+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9492082,"num_objects":1159,"num_object_clones":0,"num_object_copies":4636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":152500820,"num_objects":18621,"num_object_clones":0,"num_object_copies":74484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172086,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6760,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":305209344,"data_stored":305209344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160713,"ondisk_log_size":160713,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660984,"kb_used_data":74736,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93710856,"statfs":{"total":96636764160,"available":95959916544,"internally_reserved":0,"allocated":76529664,"data_stored":76345843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,4,0,5,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.255},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":3.2509999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":3.778},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":3.274}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":3.8159999999999998},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":3.3319999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664904,"kb_used_data":75000,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93706936,"statfs":{"total":96636764160,"available":95955902464,"internally_reserved":0,"allocated":76800000,"data_stored":76619325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,1,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.66400000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46800000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382300,"kb_used_data":76100,"kb_used_omap":4,"kb_used_meta":306171,"kb_avail":93989540,"statfs":{"total":96636764160,"available":96245288960,"internally_reserved":0,"allocated":77926400,"data_stored":77742579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":313519681},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":3.8250000000000002},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.22600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":3.7709999999999999},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":1.798}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":3.3300000000000001},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":3.6230000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669364,"kb_used_data":74860,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93702476,"statfs":{"total":96636764160,"available":95951335424,"internally_reserved":0,"allocated":76656640,"data_stored":76472819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":22.384},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":30.433}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":30.562999999999999},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":8.8360000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":15.114000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":15.077999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75931648,"data_stored":75931648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77451264,"data_stored":77451264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75821056,"data_stored":75821056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76005376,"data_stored":76005376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:00.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:02.590 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:02.609 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:02.633 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-15T03:16:00.941395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":151009876,"num_objects":18439,"num_object_clones":0,"num_object_copies":73756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172268,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6760,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2382340,"kb_used_data":297180,"kb_used_omap":14,"kb_used_meta":2085105,"kb_avail":375105020,"statfs":{"total":386547056640,"available":384107540480,"internally_reserved":0,"allocated":304312320,"data_stored":303580182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2135147547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,1,7,4,16,16,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16023552,"num_objects":-1956,"num_object_clones":0,"num_object_copies":-7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1956,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008990"},"pg_stats":[{"pgid":"1.f","version":"37'10760","reported_seq":10828,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.886368+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:15:55.886368+0000","last_peered":"2024-09-15T03:15:55.886368+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:15:55.886368+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9420800,"num_objects":1150,"num_object_clones":0,"num_object_copies":4600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10760,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10335","reported_seq":11480,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:56.029789+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:15:56.029789+0000","last_peered":"2024-09-15T03:15:56.029789+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:15:56.029789+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8935026,"num_objects":1091,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":540,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-371}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11022","reported_seq":11130,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.919759+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:15:55.919759+0000","last_peered":"2024-09-15T03:15:55.919759+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:15:55.919759+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9812817,"num_objects":1199,"num_object_clones":0,"num_object_copies":4796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1199,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11021,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-892}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10534","reported_seq":14508,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:59.305880+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:59.305880+0000","last_peered":"2024-09-15T03:15:59.305880+0000","last_clean":"2024-09-15T03:15:59.305880+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:59.305880+0000","last_undegraded":"2024-09-15T03:15:59.305880+0000","last_fullsized":"2024-09-15T03:15:59.305880+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8959602,"num_objects":1094,"num_object_clones":0,"num_object_copies":4376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10896","reported_seq":11004,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.689935+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:15:55.689935+0000","last_peered":"2024-09-15T03:15:55.689935+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:15:55.689935+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9705074,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10894,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10845","reported_seq":10931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.855696+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:15:55.855696+0000","last_peered":"2024-09-15T03:15:55.855696+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:15:55.855696+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9445376,"num_objects":1153,"num_object_clones":0,"num_object_copies":4612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10845,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10812","reported_seq":10935,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.969790+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:15:55.969790+0000","last_peered":"2024-09-15T03:15:55.969790+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:15:55.969790+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9385586,"num_objects":1146,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10919","reported_seq":11038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:56.038699+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:15:56.038699+0000","last_peered":"2024-09-15T03:15:56.038699+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:15:56.038699+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9549426,"num_objects":1166,"num_object_clones":0,"num_object_copies":4664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10917,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10782","reported_seq":10875,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.693967+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:15:55.693967+0000","last_peered":"2024-09-15T03:15:55.693967+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:15:55.693967+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9727204,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10779,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10789","reported_seq":10910,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:56.075659+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:15:56.075659+0000","last_peered":"2024-09-15T03:15:56.075659+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:15:56.075659+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9655922,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-863}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10758","reported_seq":10854,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.744883+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:15:55.744883+0000","last_peered":"2024-09-15T03:15:55.744883+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:15:55.744883+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9212331,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-937}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10764","reported_seq":14807,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:59.275171+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:59.275171+0000","last_peered":"2024-09-15T03:15:59.275171+0000","last_clean":"2024-09-15T03:15:59.275171+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:59.275171+0000","last_undegraded":"2024-09-15T03:15:59.275171+0000","last_fullsized":"2024-09-15T03:15:59.275171+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9296697,"num_objects":1135,"num_object_clones":0,"num_object_copies":4540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10763,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10968","reported_seq":15144,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:59.120450+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:59.120450+0000","last_peered":"2024-09-15T03:15:59.120450+0000","last_clean":"2024-09-15T03:15:59.120450+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:59.120450+0000","last_undegraded":"2024-09-15T03:15:59.120450+0000","last_fullsized":"2024-09-15T03:15:59.120450+0000","mapping_epoch":36,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9542457,"num_objects":1165,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10967,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10557","reported_seq":10659,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.898895+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:15:55.898895+0000","last_peered":"2024-09-15T03:15:55.898895+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:15:55.898895+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9213554,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10555,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-888}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10704","reported_seq":10813,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.956913+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:15:55.956913+0000","last_peered":"2024-09-15T03:15:55.956913+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:15:55.956913+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9655922,"num_objects":1179,"num_object_clones":0,"num_object_copies":4716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10850","reported_seq":10965,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:15:55.953729+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:15:55.953729+0000","last_peered":"2024-09-15T03:15:55.953729+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:15:55.953729+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9492082,"num_objects":1159,"num_object_clones":0,"num_object_copies":4636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10848,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":151009876,"num_objects":18439,"num_object_clones":0,"num_object_copies":73756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172268,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6760,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":301490176,"data_stored":301490176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160795,"ondisk_log_size":160795,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660984,"kb_used_data":74736,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93710856,"statfs":{"total":96636764160,"available":95959916544,"internally_reserved":0,"allocated":76529664,"data_stored":76345843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,4,0,5,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.255},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":3.2509999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":3.778},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":3.274}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":3.8159999999999998},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":3.3319999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664904,"kb_used_data":75000,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93706936,"statfs":{"total":96636764160,"available":95955902464,"internally_reserved":0,"allocated":76800000,"data_stored":76619325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,1,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.66400000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46800000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387088,"kb_used_data":72584,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93984752,"statfs":{"total":96636764160,"available":96240386048,"internally_reserved":0,"allocated":74326016,"data_stored":74142195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":8.5060000000000002},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":5.468}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.307},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":8.5470000000000006}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":1.01},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.27300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669364,"kb_used_data":74860,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93702476,"statfs":{"total":96636764160,"available":95951335424,"internally_reserved":0,"allocated":76656640,"data_stored":76472819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":22.384},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":30.433}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":30.562999999999999},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":8.8360000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":15.114000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":15.077999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75931648,"data_stored":75931648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73732096,"data_stored":73732096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75821056,"data_stored":75821056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":76005376,"data_stored":76005376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:02.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:04.331 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:04.386 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:04.431 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-15T03:16:02.941853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":145177172,"num_objects":17727,"num_object_clones":0,"num_object_copies":70908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17727,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172980,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6808,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160907,"ondisk_log_size":160907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2384080,"kb_used_data":286440,"kb_used_omap":14,"kb_used_meta":2097585,"kb_avail":375103280,"statfs":{"total":386547056640,"available":384105758720,"internally_reserved":0,"allocated":293314560,"data_stored":292582422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2147927067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,8,6,10,30,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20455424,"num_objects":-2497,"num_object_clones":0,"num_object_copies":-9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006128"},"pg_stats":[{"pgid":"1.f","version":"37'10813","reported_seq":10881,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:00.896801+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:00.896801+0000","last_peered":"2024-09-15T03:16:00.896801+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:00.896801+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8986624,"num_objects":1097,"num_object_clones":0,"num_object_copies":4388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10813,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-869}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10385","reported_seq":11626,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:01.117511+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:01.117511+0000","last_peered":"2024-09-15T03:16:01.117511+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:01.117511+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8525426,"num_objects":1041,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1414,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10383,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-373}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11082","reported_seq":11192,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:01.149042+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:01.149042+0000","last_peered":"2024-09-15T03:16:01.149042+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:01.149042+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9321297,"num_objects":1139,"num_object_clones":0,"num_object_copies":4556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1139,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11081,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-952}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10534","reported_seq":14508,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:59.305880+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:15:59.305880+0000","last_peered":"2024-09-15T03:15:59.305880+0000","last_clean":"2024-09-15T03:15:59.305880+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:15:59.305880+0000","last_undegraded":"2024-09-15T03:15:59.305880+0000","last_fullsized":"2024-09-15T03:15:59.305880+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8959602,"num_objects":1094,"num_object_clones":0,"num_object_copies":4376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'10957","reported_seq":11064,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:00.726370+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:00.726370+0000","last_peered":"2024-09-15T03:16:00.726370+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:00.726370+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9205362,"num_objects":1124,"num_object_clones":0,"num_object_copies":4496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10955,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-960}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10899","reported_seq":10985,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:00.929947+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:00.929947+0000","last_peered":"2024-09-15T03:16:00.929947+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:00.929947+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9003008,"num_objects":1099,"num_object_clones":0,"num_object_copies":4396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10899,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-939}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10864","reported_seq":10987,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:01.036513+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:01.036513+0000","last_peered":"2024-09-15T03:16:01.036513+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:01.036513+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8959602,"num_objects":1094,"num_object_clones":0,"num_object_copies":4376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10862,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-932}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'10984","reported_seq":11103,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:00.865456+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:00.865456+0000","last_peered":"2024-09-15T03:16:00.865456+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:00.865456+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9016946,"num_objects":1101,"num_object_clones":0,"num_object_copies":4404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10837","reported_seq":10930,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:00.765675+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:00.765675+0000","last_peered":"2024-09-15T03:16:00.765675+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:00.765675+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9276644,"num_objects":1133,"num_object_clones":0,"num_object_copies":4532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10834,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-894}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10842","reported_seq":10963,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:00.979947+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:00.979947+0000","last_peered":"2024-09-15T03:16:00.979947+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:00.979947+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9221746,"num_objects":1126,"num_object_clones":0,"num_object_copies":4504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10812","reported_seq":10908,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:00.892285+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:00.892285+0000","last_peered":"2024-09-15T03:16:00.892285+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:00.892285+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8769963,"num_objects":1071,"num_object_clones":0,"num_object_copies":4284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10809,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10764","reported_seq":14807,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:59.275171+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:15:59.275171+0000","last_peered":"2024-09-15T03:15:59.275171+0000","last_clean":"2024-09-15T03:15:59.275171+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:15:59.275171+0000","last_undegraded":"2024-09-15T03:15:59.275171+0000","last_fullsized":"2024-09-15T03:15:59.275171+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9296697,"num_objects":1135,"num_object_clones":0,"num_object_copies":4540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10763,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'10968","reported_seq":15144,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:15:59.120450+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:15:59.120450+0000","last_peered":"2024-09-15T03:15:59.120450+0000","last_clean":"2024-09-15T03:15:59.120450+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:15:59.120450+0000","last_undegraded":"2024-09-15T03:15:59.120450+0000","last_fullsized":"2024-09-15T03:15:59.120450+0000","mapping_epoch":36,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9542457,"num_objects":1165,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10967,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10609","reported_seq":10711,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:00.908585+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:00.908585+0000","last_peered":"2024-09-15T03:16:00.908585+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:00.908585+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8787570,"num_objects":1073,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10607,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-940}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10758","reported_seq":10866,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:01.147064+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:01.147064+0000","last_peered":"2024-09-15T03:16:01.147064+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:01.147064+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9213554,"num_objects":1125,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10756,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-852}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10899","reported_seq":11014,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:01.006161+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:01.006161+0000","last_peered":"2024-09-15T03:16:01.006161+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:01.006161+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9090674,"num_objects":1110,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10897,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-949}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":145177172,"num_objects":17727,"num_object_clones":0,"num_object_copies":70908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25876,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17727,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172980,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6808,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290365440,"data_stored":290365440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160907,"ondisk_log_size":160907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":661528,"kb_used_data":71112,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93710312,"statfs":{"total":96636764160,"available":95959359488,"internally_reserved":0,"allocated":72818688,"data_stored":72634867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,3,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.25900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":17.082000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":10.592000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.622}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665456,"kb_used_data":71400,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93706384,"statfs":{"total":96636764160,"available":95955337216,"internally_reserved":0,"allocated":73113600,"data_stored":72932925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":6.5659999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":10.968},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":11.002000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.255},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":6.6059999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":387088,"kb_used_data":72584,"kb_used_omap":4,"kb_used_meta":314491,"kb_avail":93984752,"statfs":{"total":96636764160,"available":96240386048,"internally_reserved":0,"allocated":74326016,"data_stored":74142195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":322039361},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":8.5060000000000002},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":5.468}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.307},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":8.5470000000000006}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":1.01},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.27300000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670008,"kb_used_data":71344,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93701832,"statfs":{"total":96636764160,"available":95950675968,"internally_reserved":0,"allocated":73056256,"data_stored":72872435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.441},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.65600000000000003},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.51500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":5.9809999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":72212480,"data_stored":72212480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73732096,"data_stored":73732096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":72155136,"data_stored":72155136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":72265728,"data_stored":72265728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:04.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:16:04.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10813", 'reported_seq': 10881, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:00.896801+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:16:00.896801+0000', 'last_peered': '2024-09-15T03:16:00.896801+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:16:00.896801+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8986624, 'num_objects': 1097, 'num_object_clones': 0, 'num_object_copies': 4388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10813, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -869}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:04.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:16:04.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10385", 'reported_seq': 11626, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:01.117511+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:16:01.117511+0000', 'last_peered': '2024-09-15T03:16:01.117511+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:16:01.117511+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8525426, 'num_objects': 1041, 'num_object_clones': 0, 'num_object_copies': 4164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1414, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10383, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 588, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -373}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:04.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:16:04.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11082", 'reported_seq': 11192, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:01.149042+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:16:01.149042+0000', 'last_peered': '2024-09-15T03:16:01.149042+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:16:01.149042+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9321297, 'num_objects': 1139, 'num_object_clones': 0, 'num_object_copies': 4556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1139, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11081, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -952}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'10957", 'reported_seq': 11064, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:00.726370+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:16:00.726370+0000', 'last_peered': '2024-09-15T03:16:00.726370+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:16:00.726370+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9205362, 'num_objects': 1124, 'num_object_clones': 0, 'num_object_copies': 4496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10955, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -960}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'10899", 'reported_seq': 10985, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:00.929947+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:16:00.929947+0000', 'last_peered': '2024-09-15T03:16:00.929947+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:16:00.929947+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9003008, 'num_objects': 1099, 'num_object_clones': 0, 'num_object_copies': 4396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10899, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -939}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10864", 'reported_seq': 10987, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:01.036513+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:16:01.036513+0000', 'last_peered': '2024-09-15T03:16:01.036513+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:16:01.036513+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8959602, 'num_objects': 1094, 'num_object_clones': 0, 'num_object_copies': 4376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10862, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -932}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'10984", 'reported_seq': 11103, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:00.865456+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:16:00.865456+0000', 'last_peered': '2024-09-15T03:16:00.865456+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:16:00.865456+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9016946, 'num_objects': 1101, 'num_object_clones': 0, 'num_object_copies': 4404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10982, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -955}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:16:04.434 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10837", 'reported_seq': 10930, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:00.765675+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:16:00.765675+0000', 'last_peered': '2024-09-15T03:16:00.765675+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:16:00.765675+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9276644, 'num_objects': 1133, 'num_object_clones': 0, 'num_object_copies': 4532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10834, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -894}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10842", 'reported_seq': 10963, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:00.979947+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:16:00.979947+0000', 'last_peered': '2024-09-15T03:16:00.979947+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:16:00.979947+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9221746, 'num_objects': 1126, 'num_object_clones': 0, 'num_object_copies': 4504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10840, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -916}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10812", 'reported_seq': 10908, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:00.892285+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:16:00.892285+0000', 'last_peered': '2024-09-15T03:16:00.892285+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:16:00.892285+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8769963, 'num_objects': 1071, 'num_object_clones': 0, 'num_object_copies': 4284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10809, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -991}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10609", 'reported_seq': 10711, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:00.908585+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:16:00.908585+0000', 'last_peered': '2024-09-15T03:16:00.908585+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:16:00.908585+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8787570, 'num_objects': 1073, 'num_object_clones': 0, 'num_object_copies': 4292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1073, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10607, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10758", 'reported_seq': 10866, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:01.147064+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:16:01.147064+0000', 'last_peered': '2024-09-15T03:16:01.147064+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:16:01.147064+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9213554, 'num_objects': 1125, 'num_object_clones': 0, 'num_object_copies': 4500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10756, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -852}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:16:04.435 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'10899", 'reported_seq': 11014, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:01.006161+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:16:01.006161+0000', 'last_peered': '2024-09-15T03:16:01.006161+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:16:01.006161+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9090674, 'num_objects': 1110, 'num_object_clones': 0, 'num_object_copies': 4440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10897, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:04.436 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:16:06.579 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:06.580 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25876/70200 objects degraded (36.860%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":689,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":17550,"data_bytes":143727188,"bytes_used":2424958976,"bytes_avail":384122097664,"bytes_total":386547056640,"degraded_objects":25876,"degraded_total":70200,"degraded_ratio":0.36860398860398857,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":162},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:16:06.713 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:16:06.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:08.002 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:08.009 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:08.051 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-15T03:16:06.946670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":137837140,"num_objects":16831,"num_object_clones":0,"num_object_copies":67324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173876,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6866,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2352840,"kb_used_data":271844,"kb_used_omap":14,"kb_used_meta":2080945,"kb_avail":375134520,"statfs":{"total":386547056640,"available":384137748480,"internally_reserved":0,"allocated":278368256,"data_stored":277636118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2130887707},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,8,2,3,11,15,8,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20512768,"num_objects":-2504,"num_object_clones":0,"num_object_copies":-10016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009897"},"pg_stats":[{"pgid":"1.f","version":"37'10850","reported_seq":10918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.750954+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:05.750954+0000","last_peered":"2024-09-15T03:16:05.750954+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:05.750954+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8683520,"num_objects":1060,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10850,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10436","reported_seq":11793,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.213541+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:06.213541+0000","last_peered":"2024-09-15T03:16:06.213541+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:06.213541+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8107634,"num_objects":990,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10434,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11137","reported_seq":11247,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.305115+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:06.305115+0000","last_peered":"2024-09-15T03:16:06.305115+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:06.305115+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8870737,"num_objects":1084,"num_object_clones":0,"num_object_copies":4336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11136,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10596","reported_seq":14570,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:04.155792+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:04.155792+0000","last_peered":"2024-09-15T03:16:04.155792+0000","last_clean":"2024-09-15T03:16:04.155792+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:04.155792+0000","last_undegraded":"2024-09-15T03:16:04.155792+0000","last_fullsized":"2024-09-15T03:16:04.155792+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8451698,"num_objects":1032,"num_object_clones":0,"num_object_copies":4128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10594,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11013","reported_seq":11121,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.874725+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:05.874725+0000","last_peered":"2024-09-15T03:16:05.874725+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:05.874725+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746610,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1016}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10965","reported_seq":11051,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.868507+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:05.868507+0000","last_peered":"2024-09-15T03:16:05.868507+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:05.868507+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8462336,"num_objects":1033,"num_object_clones":0,"num_object_copies":4132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10925","reported_seq":11048,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.056566+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:06.056566+0000","last_peered":"2024-09-15T03:16:06.056566+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:06.056566+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8459890,"num_objects":1033,"num_object_clones":0,"num_object_copies":4132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10923,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-993}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11049","reported_seq":11168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.950527+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:05.950527+0000","last_peered":"2024-09-15T03:16:05.950527+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:05.950527+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484466,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11047,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1020}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10889","reported_seq":10982,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.922765+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:05.922765+0000","last_peered":"2024-09-15T03:16:05.922765+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:05.922765+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8850660,"num_objects":1081,"num_object_clones":0,"num_object_copies":4324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10886,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10890","reported_seq":11011,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.207222+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:06.207222+0000","last_peered":"2024-09-15T03:16:06.207222+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:06.207222+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8828530,"num_objects":1078,"num_object_clones":0,"num_object_copies":4312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10888,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10858","reported_seq":10954,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.029124+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:06.029124+0000","last_peered":"2024-09-15T03:16:06.029124+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:06.029124+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8393131,"num_objects":1025,"num_object_clones":0,"num_object_copies":4100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10855,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10829","reported_seq":14870,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:04.036936+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:04.036936+0000","last_peered":"2024-09-15T03:16:04.036936+0000","last_clean":"2024-09-15T03:16:04.036936+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:04.036936+0000","last_undegraded":"2024-09-15T03:16:04.036936+0000","last_fullsized":"2024-09-15T03:16:04.036936+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8764217,"num_objects":1070,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11018","reported_seq":15194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:04.262595+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:04.262595+0000","last_peered":"2024-09-15T03:16:04.262595+0000","last_clean":"2024-09-15T03:16:04.262595+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:04.262595+0000","last_undegraded":"2024-09-15T03:16:04.262595+0000","last_fullsized":"2024-09-15T03:16:04.262595+0000","mapping_epoch":36,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9132857,"num_objects":1115,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11017,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10665","reported_seq":10767,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.720629+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:05.720629+0000","last_peered":"2024-09-15T03:16:05.720629+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:05.720629+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8328818,"num_objects":1017,"num_object_clones":0,"num_object_copies":4068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10824","reported_seq":10932,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.240557+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:06.240557+0000","last_peered":"2024-09-15T03:16:06.240557+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:06.240557+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672882,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10822,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10959","reported_seq":11073,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.092867+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:06.092867+0000","last_peered":"2024-09-15T03:16:06.092867+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:06.092867+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8599154,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10957,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1009}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":137837140,"num_objects":16831,"num_object_clones":0,"num_object_copies":67324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173876,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6866,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275910656,"data_stored":275910656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662164,"kb_used_data":67596,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93709676,"statfs":{"total":96636764160,"available":95958708224,"internally_reserved":0,"allocated":69218304,"data_stored":69034483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,7,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.36699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":7.0670000000000002},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.25700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":7.0199999999999996},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.28199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665448,"kb_used_data":67228,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93706392,"statfs":{"total":96636764160,"available":95955345408,"internally_reserved":0,"allocated":68841472,"data_stored":68660797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":5.0490000000000004},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.309}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.42699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.34100000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371132,"kb_used_data":69108,"kb_used_omap":4,"kb_used_meta":302011,"kb_avail":94000708,"statfs":{"total":96636764160,"available":96256724992,"internally_reserved":0,"allocated":70766592,"data_stored":70582771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":309259841},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,0,2,0,1,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":5.4189999999999996},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":13.845000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":5.6219999999999999},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":5.4500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":13.766999999999999},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":5.6639999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654096,"kb_used_data":67912,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93717744,"statfs":{"total":96636764160,"available":95966969856,"internally_reserved":0,"allocated":69541888,"data_stored":69358067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":8.0069999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":4.2229999999999999},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.38200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":68595712,"data_stored":68595712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":70135808,"data_stored":70135808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":68493312,"data_stored":68493312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":68685824,"data_stored":68685824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:11.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:12.850 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:12.889 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:12.956 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-15T03:16:10.947635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":136485460,"num_objects":16666,"num_object_clones":0,"num_object_copies":66664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174041,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6866,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160868,"ondisk_log_size":160868,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2641964,"kb_used_data":268416,"kb_used_omap":14,"kb_used_meta":2373489,"kb_avail":374845396,"statfs":{"total":386547056640,"available":383841685504,"internally_reserved":0,"allocated":274857984,"data_stored":274125846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430452763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,2,2,12,19,7,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16015360,"num_objects":-1955,"num_object_clones":0,"num_object_copies":-7820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1955,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006694"},"pg_stats":[{"pgid":"1.f","version":"37'10850","reported_seq":10918,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.750954+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:05.750954+0000","last_peered":"2024-09-15T03:16:05.750954+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:05.750954+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8683520,"num_objects":1060,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10850,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10436","reported_seq":11793,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.213541+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:06.213541+0000","last_peered":"2024-09-15T03:16:06.213541+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:06.213541+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8107634,"num_objects":990,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10434,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":646,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11137","reported_seq":11247,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.305115+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:06.305115+0000","last_peered":"2024-09-15T03:16:06.305115+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:06.305115+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8870737,"num_objects":1084,"num_object_clones":0,"num_object_copies":4336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11136,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10650","reported_seq":14624,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:09.140780+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:09.140780+0000","last_peered":"2024-09-15T03:16:09.140780+0000","last_clean":"2024-09-15T03:16:09.140780+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:09.140780+0000","last_undegraded":"2024-09-15T03:16:09.140780+0000","last_fullsized":"2024-09-15T03:16:09.140780+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8009330,"num_objects":978,"num_object_clones":0,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10648,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11013","reported_seq":11121,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.874725+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:05.874725+0000","last_peered":"2024-09-15T03:16:05.874725+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:05.874725+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746610,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11011,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1016}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'10965","reported_seq":11051,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.868507+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:05.868507+0000","last_peered":"2024-09-15T03:16:05.868507+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:05.868507+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8462336,"num_objects":1033,"num_object_clones":0,"num_object_copies":4132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10965,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1005}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10925","reported_seq":11048,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.056566+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:06.056566+0000","last_peered":"2024-09-15T03:16:06.056566+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:06.056566+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8459890,"num_objects":1033,"num_object_clones":0,"num_object_copies":4132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10923,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-993}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11049","reported_seq":11168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.950527+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:05.950527+0000","last_peered":"2024-09-15T03:16:05.950527+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:05.950527+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484466,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11047,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1020}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10889","reported_seq":10982,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.922765+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:05.922765+0000","last_peered":"2024-09-15T03:16:05.922765+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:05.922765+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8850660,"num_objects":1081,"num_object_clones":0,"num_object_copies":4324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10886,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10890","reported_seq":11011,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.207222+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:06.207222+0000","last_peered":"2024-09-15T03:16:06.207222+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:06.207222+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8828530,"num_objects":1078,"num_object_clones":0,"num_object_copies":4312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10888,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10858","reported_seq":10954,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.029124+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:06.029124+0000","last_peered":"2024-09-15T03:16:06.029124+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:06.029124+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8393131,"num_objects":1025,"num_object_clones":0,"num_object_copies":4100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10855,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1037}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10885","reported_seq":14928,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:09.322180+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:09.322180+0000","last_peered":"2024-09-15T03:16:09.322180+0000","last_clean":"2024-09-15T03:16:09.322180+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:09.322180+0000","last_undegraded":"2024-09-15T03:16:09.322180+0000","last_fullsized":"2024-09-15T03:16:09.322180+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8305465,"num_objects":1014,"num_object_clones":0,"num_object_copies":4056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10884,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11073","reported_seq":15249,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:09.184606+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:09.184606+0000","last_peered":"2024-09-15T03:16:09.184606+0000","last_clean":"2024-09-15T03:16:09.184606+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:09.184606+0000","last_undegraded":"2024-09-15T03:16:09.184606+0000","last_fullsized":"2024-09-15T03:16:09.184606+0000","mapping_epoch":36,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8682297,"num_objects":1060,"num_object_clones":0,"num_object_copies":4240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11072,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10665","reported_seq":10767,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:05.720629+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:05.720629+0000","last_peered":"2024-09-15T03:16:05.720629+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:05.720629+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8328818,"num_objects":1017,"num_object_clones":0,"num_object_copies":4068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-996}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10824","reported_seq":10932,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.240557+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:06.240557+0000","last_peered":"2024-09-15T03:16:06.240557+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:06.240557+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672882,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10822,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-918}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'10959","reported_seq":11073,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:06.092867+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:06.092867+0000","last_peered":"2024-09-15T03:16:06.092867+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:06.092867+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8599154,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10957,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1009}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":136485460,"num_objects":16666,"num_object_clones":0,"num_object_copies":66664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25818,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174041,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6866,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272265216,"data_stored":272265216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160868,"ondisk_log_size":160868,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662164,"kb_used_data":67596,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93709676,"statfs":{"total":96636764160,"available":95958708224,"internally_reserved":0,"allocated":69218304,"data_stored":69034483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,7,1,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.36699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":7.0670000000000002},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.25700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":7.0199999999999996},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.28199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665448,"kb_used_data":67228,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93706392,"statfs":{"total":96636764160,"available":95955345408,"internally_reserved":0,"allocated":68841472,"data_stored":68660797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":5.0490000000000004},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.309}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.42699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.34100000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660256,"kb_used_data":65680,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93711584,"statfs":{"total":96636764160,"available":95960662016,"internally_reserved":0,"allocated":67256320,"data_stored":67072499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":5.4989999999999997},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":5.2949999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":5.1399999999999997},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":1.2050000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":12.875},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":5.1070000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654096,"kb_used_data":67912,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93717744,"statfs":{"total":96636764160,"available":95966969856,"internally_reserved":0,"allocated":69541888,"data_stored":69358067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":8.0069999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":4.2229999999999999},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.38200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.32500000000000001},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.35499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":68595712,"data_stored":68595712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66490368,"data_stored":66490368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":68493312,"data_stored":68493312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":68685824,"data_stored":68685824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:12.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:15.080 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:15.112 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:15.158 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-15T03:16:14.952448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":129055316,"num_objects":15759,"num_object_clones":0,"num_object_copies":63036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174948,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6942,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2628192,"kb_used_data":254648,"kb_used_omap":14,"kb_used_meta":2373489,"kb_avail":374859168,"statfs":{"total":386547056640,"available":383855788032,"internally_reserved":0,"allocated":260759552,"data_stored":260027414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430452763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,8,7,16,14,9,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16121856,"num_objects":-1968,"num_object_clones":0,"num_object_copies":-7872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010594"},"pg_stats":[{"pgid":"1.f","version":"37'10908","reported_seq":10976,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:10.877746+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:10.877746+0000","last_peered":"2024-09-15T03:16:10.877746+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:10.877746+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8208384,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10908,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-964}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10483","reported_seq":11989,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:11.287849+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:11.287849+0000","last_peered":"2024-09-15T03:16:11.287849+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:11.287849+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7722610,"num_objects":943,"num_object_clones":0,"num_object_copies":3772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10481,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":722,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11188","reported_seq":11297,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:11.293351+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:11.293351+0000","last_peered":"2024-09-15T03:16:11.293351+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:11.293351+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8452945,"num_objects":1033,"num_object_clones":0,"num_object_copies":4132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11187,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1058}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10703","reported_seq":14677,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:14.307414+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:14.307414+0000","last_peered":"2024-09-15T03:16:14.307414+0000","last_clean":"2024-09-15T03:16:14.307414+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:14.307414+0000","last_undegraded":"2024-09-15T03:16:14.307414+0000","last_fullsized":"2024-09-15T03:16:14.307414+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7575154,"num_objects":925,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10701,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11080","reported_seq":11188,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:11.074470+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:11.074470+0000","last_peered":"2024-09-15T03:16:11.074470+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:11.074470+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8197746,"num_objects":1001,"num_object_clones":0,"num_object_copies":4004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11078,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11020","reported_seq":11106,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:10.972952+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:10.972952+0000","last_peered":"2024-09-15T03:16:10.972952+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:10.972952+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8011776,"num_objects":978,"num_object_clones":0,"num_object_copies":3912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1060}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'10973","reported_seq":11096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:11.194729+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:11.194729+0000","last_peered":"2024-09-15T03:16:11.194729+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:11.194729+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8066674,"num_objects":985,"num_object_clones":0,"num_object_copies":3940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10971,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1041}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11113","reported_seq":11232,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:11.178484+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:11.178484+0000","last_peered":"2024-09-15T03:16:11.178484+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:11.178484+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7960178,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1084}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10955","reported_seq":11048,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:10.997508+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:10.997508+0000","last_peered":"2024-09-15T03:16:10.997508+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:10.997508+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8309988,"num_objects":1015,"num_object_clones":0,"num_object_copies":4060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10952,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1012}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'10949","reported_seq":11070,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:11.293315+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:11.293315+0000","last_peered":"2024-09-15T03:16:11.293315+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:11.293315+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8345202,"num_objects":1019,"num_object_clones":0,"num_object_copies":4076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10947,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1023}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10898","reported_seq":10993,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:10.932665+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:10.932665+0000","last_peered":"2024-09-15T03:16:10.932665+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:10.932665+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8065451,"num_objects":985,"num_object_clones":0,"num_object_copies":3940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10895,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1077}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10947","reported_seq":14990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:14.365371+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:14.365371+0000","last_peered":"2024-09-15T03:16:14.365371+0000","last_clean":"2024-09-15T03:16:14.365371+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:14.365371+0000","last_undegraded":"2024-09-15T03:16:14.365371+0000","last_fullsized":"2024-09-15T03:16:14.365371+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7797561,"num_objects":952,"num_object_clones":0,"num_object_copies":3808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10946,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11139","reported_seq":15314,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:14.525160+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:14.525160+0000","last_peered":"2024-09-15T03:16:14.525160+0000","last_clean":"2024-09-15T03:16:14.525160+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:14.525160+0000","last_undegraded":"2024-09-15T03:16:14.525160+0000","last_fullsized":"2024-09-15T03:16:14.525160+0000","mapping_epoch":36,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8141625,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11138,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10715","reported_seq":10817,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:10.976950+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:10.976950+0000","last_peered":"2024-09-15T03:16:10.976950+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:10.976950+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7919218,"num_objects":967,"num_object_clones":0,"num_object_copies":3868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10713,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1046}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10887","reported_seq":10996,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:11.194853+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:11.194853+0000","last_peered":"2024-09-15T03:16:11.194853+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:11.194853+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8156786,"num_objects":996,"num_object_clones":0,"num_object_copies":3984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10885,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-981}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11017","reported_seq":11132,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:10.866278+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:10.866278+0000","last_peered":"2024-09-15T03:16:10.866278+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:10.866278+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8124018,"num_objects":992,"num_object_clones":0,"num_object_copies":3968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11015,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1067}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":129055316,"num_objects":15759,"num_object_clones":0,"num_object_copies":63036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15759,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":174948,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6942,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":257581056,"data_stored":257581056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":662644,"kb_used_data":63912,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93709196,"statfs":{"total":96636764160,"available":95958216704,"internally_reserved":0,"allocated":65445888,"data_stored":65262067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,4,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.30499999999999999},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.27800000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":2.6440000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":2.6280000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.33800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650068,"kb_used_data":64332,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93721772,"statfs":{"total":96636764160,"available":95971094528,"internally_reserved":0,"allocated":65875968,"data_stored":65695293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.28199999999999997},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.34899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":8.6270000000000007},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":8.6600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":2.746},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.38200000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660784,"kb_used_data":62052,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93711056,"statfs":{"total":96636764160,"available":95960121344,"internally_reserved":0,"allocated":63541248,"data_stored":63357427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,4,5,8,4,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.193},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":0.52200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":2.4500000000000002},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":2.4670000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":654696,"kb_used_data":64352,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93717144,"statfs":{"total":96636764160,"available":95966355456,"internally_reserved":0,"allocated":65896448,"data_stored":65712627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.29899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.34000000000000002},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.57099999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.314},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":64921600,"data_stored":64921600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62799872,"data_stored":62799872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":64851968,"data_stored":64851968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65007616,"data_stored":65007616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:15.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'10908", 'reported_seq': 10976, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:10.877746+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:16:10.877746+0000', 'last_peered': '2024-09-15T03:16:10.877746+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:16:10.877746+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8208384, 'num_objects': 1002, 'num_object_clones': 0, 'num_object_copies': 4008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10908, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -964}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10483", 'reported_seq': 11989, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:11.287849+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:16:11.287849+0000', 'last_peered': '2024-09-15T03:16:11.287849+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:16:11.287849+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7722610, 'num_objects': 943, 'num_object_clones': 0, 'num_object_copies': 3772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10481, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 722, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11188", 'reported_seq': 11297, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:11.293351+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:16:11.293351+0000', 'last_peered': '2024-09-15T03:16:11.293351+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:16:11.293351+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8452945, 'num_objects': 1033, 'num_object_clones': 0, 'num_object_copies': 4132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1033, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11187, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1058}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11080", 'reported_seq': 11188, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:11.074470+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:16:11.074470+0000', 'last_peered': '2024-09-15T03:16:11.074470+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:16:11.074470+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8197746, 'num_objects': 1001, 'num_object_clones': 0, 'num_object_copies': 4004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11078, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1083}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:16:15.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11020", 'reported_seq': 11106, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:10.972952+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:16:10.972952+0000', 'last_peered': '2024-09-15T03:16:10.972952+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:16:10.972952+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8011776, 'num_objects': 978, 'num_object_clones': 0, 'num_object_copies': 3912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11020, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1060}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'10973", 'reported_seq': 11096, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:11.194729+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:16:11.194729+0000', 'last_peered': '2024-09-15T03:16:11.194729+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:16:11.194729+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8066674, 'num_objects': 985, 'num_object_clones': 0, 'num_object_copies': 3940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10971, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1041}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11113", 'reported_seq': 11232, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:11.178484+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:16:11.178484+0000', 'last_peered': '2024-09-15T03:16:11.178484+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:16:11.178484+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7960178, 'num_objects': 972, 'num_object_clones': 0, 'num_object_copies': 3888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11111, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1084}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'10955", 'reported_seq': 11048, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:10.997508+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:16:10.997508+0000', 'last_peered': '2024-09-15T03:16:10.997508+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:16:10.997508+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8309988, 'num_objects': 1015, 'num_object_clones': 0, 'num_object_copies': 4060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10952, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1012}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'10949", 'reported_seq': 11070, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:11.293315+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:16:11.293315+0000', 'last_peered': '2024-09-15T03:16:11.293315+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:16:11.293315+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8345202, 'num_objects': 1019, 'num_object_clones': 0, 'num_object_copies': 4076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10947, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1023}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:15.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:16:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'10898", 'reported_seq': 10993, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:10.932665+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:16:10.932665+0000', 'last_peered': '2024-09-15T03:16:10.932665+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:16:10.932665+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8065451, 'num_objects': 985, 'num_object_clones': 0, 'num_object_copies': 3940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10895, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1077}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:16:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10715", 'reported_seq': 10817, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:10.976950+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:16:10.976950+0000', 'last_peered': '2024-09-15T03:16:10.976950+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:16:10.976950+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7919218, 'num_objects': 967, 'num_object_clones': 0, 'num_object_copies': 3868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10713, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1046}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:16:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10887", 'reported_seq': 10996, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:11.194853+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:16:11.194853+0000', 'last_peered': '2024-09-15T03:16:11.194853+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:16:11.194853+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8156786, 'num_objects': 996, 'num_object_clones': 0, 'num_object_copies': 3984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10885, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -981}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:16:15.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11017", 'reported_seq': 11132, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:10.866278+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:16:10.866278+0000', 'last_peered': '2024-09-15T03:16:10.866278+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:16:10.866278+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8124018, 'num_objects': 992, 'num_object_clones': 0, 'num_object_copies': 3968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11015, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1067}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:15.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:16:17.456 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:17.456 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25692/60212 objects degraded (42.669%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":700,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":15053,"data_bytes":123271764,"bytes_used":2680344576,"bytes_avail":383866712064,"bytes_total":386547056640,"degraded_objects":25692,"degraded_total":60212,"degraded_ratio":0.4266923536836511,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:16:17.543 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:16:17.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:19.371 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:19.471 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:19.503 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-15T03:16:18.953452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123271764,"num_objects":15053,"num_object_clones":0,"num_object_copies":60212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175654,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6992,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2617524,"kb_used_data":243984,"kb_used_omap":14,"kb_used_meta":2373489,"kb_avail":374869836,"statfs":{"total":386547056640,"available":383866712064,"internally_reserved":0,"allocated":249839616,"data_stored":249107478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430452763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,5,1,2,10,13,13,8,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14565376,"num_objects":-1778,"num_object_clones":0,"num_object_copies":-7112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1778,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006781"},"pg_stats":[{"pgid":"1.f","version":"37'10959","reported_seq":11027,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.010464+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:16.010464+0000","last_peered":"2024-09-15T03:16:16.010464+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:16.010464+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7790592,"num_objects":951,"num_object_clones":0,"num_object_copies":3804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10959,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1015}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10534","reported_seq":12143,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.202392+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:16.202392+0000","last_peered":"2024-09-15T03:16:16.202392+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:16.202392+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7304818,"num_objects":892,"num_object_clones":0,"num_object_copies":3568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1230,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":772,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-338}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11239","reported_seq":11349,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.167626+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:16.167626+0000","last_peered":"2024-09-15T03:16:16.167626+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:16.167626+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8035153,"num_objects":982,"num_object_clones":0,"num_object_copies":3928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11238,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10703","reported_seq":14677,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:14.307414+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:14.307414+0000","last_peered":"2024-09-15T03:16:14.307414+0000","last_clean":"2024-09-15T03:16:14.307414+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:14.307414+0000","last_undegraded":"2024-09-15T03:16:14.307414+0000","last_fullsized":"2024-09-15T03:16:14.307414+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7575154,"num_objects":925,"num_object_clones":0,"num_object_copies":3700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10701,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11155","reported_seq":11263,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.166074+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:16.166074+0000","last_peered":"2024-09-15T03:16:16.166074+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:16.166074+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7583346,"num_objects":926,"num_object_clones":0,"num_object_copies":3704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11153,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1158}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11082","reported_seq":11168,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.202613+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:16.202613+0000","last_peered":"2024-09-15T03:16:16.202613+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:16.202613+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7503872,"num_objects":916,"num_object_clones":0,"num_object_copies":3664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11030","reported_seq":11153,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.122094+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:16.122094+0000","last_peered":"2024-09-15T03:16:16.122094+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:16.122094+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7599730,"num_objects":928,"num_object_clones":0,"num_object_copies":3712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11028,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1098}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11170","reported_seq":11289,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.160160+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:16.160160+0000","last_peered":"2024-09-15T03:16:16.160160+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:16.160160+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7493234,"num_objects":915,"num_object_clones":0,"num_object_copies":3660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'10998","reported_seq":11091,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.255307+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:16.255307+0000","last_peered":"2024-09-15T03:16:16.255307+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:16.255307+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7957732,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10995,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1055}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11005","reported_seq":11126,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.311634+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:16.311634+0000","last_peered":"2024-09-15T03:16:16.311634+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:16.311634+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7886450,"num_objects":963,"num_object_clones":0,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11003,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1079}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'10945","reported_seq":11041,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:15.820775+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:15.820775+0000","last_peered":"2024-09-15T03:16:15.820775+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:15.820775+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7680427,"num_objects":938,"num_object_clones":0,"num_object_copies":3752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10942,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1124}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'10947","reported_seq":14990,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:14.365371+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:14.365371+0000","last_peered":"2024-09-15T03:16:14.365371+0000","last_clean":"2024-09-15T03:16:14.365371+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:14.365371+0000","last_undegraded":"2024-09-15T03:16:14.365371+0000","last_fullsized":"2024-09-15T03:16:14.365371+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7797561,"num_objects":952,"num_object_clones":0,"num_object_copies":3808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10946,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11139","reported_seq":15314,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:14.525160+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:14.525160+0000","last_peered":"2024-09-15T03:16:14.525160+0000","last_clean":"2024-09-15T03:16:14.525160+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:14.525160+0000","last_undegraded":"2024-09-15T03:16:14.525160+0000","last_fullsized":"2024-09-15T03:16:14.525160+0000","mapping_epoch":36,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8141625,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11138,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10760","reported_seq":10862,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.076038+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:16.076038+0000","last_peered":"2024-09-15T03:16:16.076038+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:16.076038+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7550578,"num_objects":922,"num_object_clones":0,"num_object_copies":3688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10758,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1091}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10938","reported_seq":11047,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.358299+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:16.358299+0000","last_peered":"2024-09-15T03:16:16.358299+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:16.358299+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7738994,"num_objects":945,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1032}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11077","reported_seq":11190,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:16.059319+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:16.059319+0000","last_peered":"2024-09-15T03:16:16.059319+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:16.059319+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7632498,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11075,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1127}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":123271764,"num_objects":15053,"num_object_clones":0,"num_object_copies":60212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15053,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175654,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6992,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":246685696,"data_stored":246685696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650800,"kb_used_data":60392,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93721040,"statfs":{"total":96636764160,"available":95970344960,"internally_reserved":0,"allocated":61841408,"data_stored":61657587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,0,1,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":2.4700000000000002},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":2.1840000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":2.4060000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":2.4449999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650640,"kb_used_data":60744,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93721200,"statfs":{"total":96636764160,"available":95970508800,"internally_reserved":0,"allocated":62201856,"data_stored":62021181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":13.526999999999999},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.67100000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.376},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.40899999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.70399999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660784,"kb_used_data":62052,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93711056,"statfs":{"total":96636764160,"available":95960121344,"internally_reserved":0,"allocated":63541248,"data_stored":63357427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,2,4,5,8,4,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.3209999999999997,"5min":3.286,"15min":2.3090000000000002},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":20.346,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.193},{"interface":"front","average":{"1min":3.3319999999999999,"5min":2.6560000000000001,"15min":1.927},"min":{"1min":0.21299999999999999,"5min":0.191,"15min":0.187},"max":{"1min":35.463000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":4.8109999999999999,"5min":3.1960000000000002,"15min":2.0609999999999999},"min":{"1min":0.23200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":25.661000000000001,"5min":26.911000000000001,"15min":26.911000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":4.3680000000000003,"5min":4.4850000000000003,"15min":2.6389999999999998},"min":{"1min":0.31,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":17.347999999999999,"5min":28.145,"15min":28.145},"last":0.52200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:13 2024","interfaces":[{"interface":"back","average":{"1min":5.907,"5min":3.4580000000000002,"15min":2.21},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.222},"max":{"1min":34.875,"5min":59.225999999999999,"15min":59.225999999999999},"last":2.4500000000000002},{"interface":"front","average":{"1min":3.298,"5min":2.3799999999999999,"15min":1.6579999999999999},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":33.444000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":2.4670000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655300,"kb_used_data":60796,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93716540,"statfs":{"total":96636764160,"available":95965736960,"internally_reserved":0,"allocated":62255104,"data_stored":62071283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.189},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":3.2810000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":3.234},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.32700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61308928,"data_stored":61308928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62799872,"data_stored":62799872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61247488,"data_stored":61247488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61329408,"data_stored":61329408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:22.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:24.415 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:24.492 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:24.548 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-15T03:16:22.954375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115776084,"num_objects":14138,"num_object_clones":0,"num_object_copies":56552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14138,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176569,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7036,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160896,"ondisk_log_size":160896,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2607452,"kb_used_data":229728,"kb_used_omap":14,"kb_used_meta":2377649,"kb_avail":374879908,"statfs":{"total":386547056640,"available":383877025792,"internally_reserved":0,"allocated":235241472,"data_stored":234509334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2434712603},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,0,20,11,6,15,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20709376,"num_objects":-2528,"num_object_clones":0,"num_object_copies":-10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006740"},"pg_stats":[{"pgid":"1.f","version":"37'11006","reported_seq":11074,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.047520+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:21.047520+0000","last_peered":"2024-09-15T03:16:21.047520+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:21.047520+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":904,"num_object_clones":0,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11006,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10589","reported_seq":12286,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.176291+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:21.176291+0000","last_peered":"2024-09-15T03:16:21.176291+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:21.176291+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6854258,"num_objects":837,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11304","reported_seq":11414,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.282854+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:21.282854+0000","last_peered":"2024-09-15T03:16:21.282854+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:21.282854+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7502673,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11303,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1174}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10764","reported_seq":14738,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:19.596548+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:19.596548+0000","last_peered":"2024-09-15T03:16:19.596548+0000","last_clean":"2024-09-15T03:16:19.596548+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:19.596548+0000","last_undegraded":"2024-09-15T03:16:19.596548+0000","last_fullsized":"2024-09-15T03:16:19.596548+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075442,"num_objects":864,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10762,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11198","reported_seq":11306,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.319748+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:21.319748+0000","last_peered":"2024-09-15T03:16:21.319748+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:21.319748+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7231090,"num_objects":883,"num_object_clones":0,"num_object_copies":3532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11150","reported_seq":11236,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.120596+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:21.120596+0000","last_peered":"2024-09-15T03:16:21.120596+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:21.120596+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6946816,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11150,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11104","reported_seq":11226,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.206253+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:21.206253+0000","last_peered":"2024-09-15T03:16:21.206253+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:21.206253+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6993522,"num_objects":854,"num_object_clones":0,"num_object_copies":3416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11102,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1172}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11224","reported_seq":11343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.021091+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:21.021091+0000","last_peered":"2024-09-15T03:16:21.021091+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:21.021091+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7050866,"num_objects":861,"num_object_clones":0,"num_object_copies":3444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1195}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11053","reported_seq":11146,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.216742+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:21.216742+0000","last_peered":"2024-09-15T03:16:21.216742+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:21.216742+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7507172,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1110}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11059","reported_seq":11180,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.277172+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:21.277172+0000","last_peered":"2024-09-15T03:16:21.277172+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:21.277172+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7444082,"num_objects":909,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1133}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11000","reported_seq":11096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.181714+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:21.181714+0000","last_peered":"2024-09-15T03:16:21.181714+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:21.181714+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7229867,"num_objects":883,"num_object_clones":0,"num_object_copies":3532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11000","reported_seq":15043,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:19.256121+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:19.256121+0000","last_peered":"2024-09-15T03:16:19.256121+0000","last_clean":"2024-09-15T03:16:19.256121+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:19.256121+0000","last_undegraded":"2024-09-15T03:16:19.256121+0000","last_fullsized":"2024-09-15T03:16:19.256121+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7363385,"num_objects":899,"num_object_clones":0,"num_object_copies":3596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10999,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11188","reported_seq":15364,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:19.520109+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:19.520109+0000","last_peered":"2024-09-15T03:16:19.520109+0000","last_clean":"2024-09-15T03:16:19.520109+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:19.520109+0000","last_undegraded":"2024-09-15T03:16:19.520109+0000","last_fullsized":"2024-09-15T03:16:19.520109+0000","mapping_epoch":36,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7740217,"num_objects":945,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10831","reported_seq":10933,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.257711+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:21.257711+0000","last_peered":"2024-09-15T03:16:21.257711+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:21.257711+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6968946,"num_objects":851,"num_object_clones":0,"num_object_copies":3404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10996","reported_seq":11105,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.205060+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:21.205060+0000","last_peered":"2024-09-15T03:16:21.205060+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:21.205060+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7263858,"num_objects":887,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10994,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1090}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11130","reported_seq":11242,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.217191+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:21.217191+0000","last_peered":"2024-09-15T03:16:21.217191+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:21.217191+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7198322,"num_objects":879,"num_object_clones":0,"num_object_copies":3516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":115776084,"num_objects":14138,"num_object_clones":0,"num_object_copies":56552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14138,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176569,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7036,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":231927808,"data_stored":231927808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160896,"ondisk_log_size":160896,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651404,"kb_used_data":56836,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93720436,"statfs":{"total":96636764160,"available":95969726464,"internally_reserved":0,"allocated":58200064,"data_stored":58016243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":1.8069999999999999},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.5860000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.36799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.747},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.32700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655392,"kb_used_data":57168,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93716448,"statfs":{"total":96636764160,"available":95965642752,"internally_reserved":0,"allocated":58540032,"data_stored":58359357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,0,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.314}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.3959999999999999},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":3.4249999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":7.7789999999999999},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":13.07}]}]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644764,"kb_used_data":58496,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93727076,"statfs":{"total":96636764160,"available":95976525824,"internally_reserved":0,"allocated":59899904,"data_stored":59716083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,4,3,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":4.5469999999999997},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.28799999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":8.3100000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":4.5919999999999996},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.3959999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655892,"kb_used_data":57228,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93715948,"statfs":{"total":96636764160,"available":95965130752,"internally_reserved":0,"allocated":58601472,"data_stored":58417651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,13,4,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.42299999999999999},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.35299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57634816,"data_stored":57634816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59121664,"data_stored":59121664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57548800,"data_stored":57548800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57622528,"data_stored":57622528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:24.550 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:26.482 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:26.502 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:26.594 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-15T03:16:24.954896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114383444,"num_objects":13968,"num_object_clones":0,"num_object_copies":55872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176739,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7036,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2612172,"kb_used_data":226148,"kb_used_omap":14,"kb_used_meta":2385969,"kb_avail":374875188,"statfs":{"total":386547056640,"available":383872192512,"internally_reserved":0,"allocated":231575552,"data_stored":230843414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2443232283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,4,1,17,14,9,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16154624,"num_objects":-1972,"num_object_clones":0,"num_object_copies":-7888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1972,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005167"},"pg_stats":[{"pgid":"1.f","version":"37'11006","reported_seq":11074,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.047520+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:21.047520+0000","last_peered":"2024-09-15T03:16:21.047520+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:21.047520+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":904,"num_object_clones":0,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11006,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10589","reported_seq":12286,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.176291+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:21.176291+0000","last_peered":"2024-09-15T03:16:21.176291+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:21.176291+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6854258,"num_objects":837,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10587,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11304","reported_seq":11414,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.282854+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:21.282854+0000","last_peered":"2024-09-15T03:16:21.282854+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:21.282854+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7502673,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11303,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1174}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10833","reported_seq":14807,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:24.541634+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:24.541634+0000","last_peered":"2024-09-15T03:16:24.541634+0000","last_clean":"2024-09-15T03:16:24.541634+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:24.541634+0000","last_undegraded":"2024-09-15T03:16:24.541634+0000","last_fullsized":"2024-09-15T03:16:24.541634+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6510194,"num_objects":795,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10831,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11198","reported_seq":11306,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.319748+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:21.319748+0000","last_peered":"2024-09-15T03:16:21.319748+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:21.319748+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7231090,"num_objects":883,"num_object_clones":0,"num_object_copies":3532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11150","reported_seq":11236,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.120596+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:21.120596+0000","last_peered":"2024-09-15T03:16:21.120596+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:21.120596+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6946816,"num_objects":848,"num_object_clones":0,"num_object_copies":3392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11150,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1190}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11104","reported_seq":11226,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.206253+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:21.206253+0000","last_peered":"2024-09-15T03:16:21.206253+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:21.206253+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6993522,"num_objects":854,"num_object_clones":0,"num_object_copies":3416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11102,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1172}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11224","reported_seq":11343,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.021091+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:21.021091+0000","last_peered":"2024-09-15T03:16:21.021091+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:21.021091+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7050866,"num_objects":861,"num_object_clones":0,"num_object_copies":3444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1195}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11053","reported_seq":11146,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.216742+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:21.216742+0000","last_peered":"2024-09-15T03:16:21.216742+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:21.216742+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7507172,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1110}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11059","reported_seq":11180,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.277172+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:21.277172+0000","last_peered":"2024-09-15T03:16:21.277172+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:21.277172+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7444082,"num_objects":909,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1133}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11000","reported_seq":11096,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.181714+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:21.181714+0000","last_peered":"2024-09-15T03:16:21.181714+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:21.181714+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7229867,"num_objects":883,"num_object_clones":0,"num_object_copies":3532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11052","reported_seq":15095,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:24.561253+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:24.561253+0000","last_peered":"2024-09-15T03:16:24.561253+0000","last_clean":"2024-09-15T03:16:24.561253+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:24.561253+0000","last_undegraded":"2024-09-15T03:16:24.561253+0000","last_fullsized":"2024-09-15T03:16:24.561253+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6937401,"num_objects":847,"num_object_clones":0,"num_object_copies":3388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11237","reported_seq":15413,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:24.356492+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:24.356492+0000","last_peered":"2024-09-15T03:16:24.356492+0000","last_clean":"2024-09-15T03:16:24.356492+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:24.356492+0000","last_undegraded":"2024-09-15T03:16:24.356492+0000","last_fullsized":"2024-09-15T03:16:24.356492+0000","mapping_epoch":36,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7338809,"num_objects":896,"num_object_clones":0,"num_object_copies":3584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11236,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10831","reported_seq":10933,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.257711+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:21.257711+0000","last_peered":"2024-09-15T03:16:21.257711+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:21.257711+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6968946,"num_objects":851,"num_object_clones":0,"num_object_copies":3404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1162}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'10996","reported_seq":11105,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.205060+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:21.205060+0000","last_peered":"2024-09-15T03:16:21.205060+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:21.205060+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7263858,"num_objects":887,"num_object_clones":0,"num_object_copies":3548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10994,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1090}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11130","reported_seq":11242,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:21.217191+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:21.217191+0000","last_peered":"2024-09-15T03:16:21.217191+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:21.217191+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7198322,"num_objects":879,"num_object_clones":0,"num_object_copies":3516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11128,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1180}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":114383444,"num_objects":13968,"num_object_clones":0,"num_object_copies":55872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":176739,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7036,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":228315136,"data_stored":228315136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160766,"ondisk_log_size":160766,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651404,"kb_used_data":56836,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93720436,"statfs":{"total":96636764160,"available":95969726464,"internally_reserved":0,"allocated":58200064,"data_stored":58016243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":1.8069999999999999},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":1.5860000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.36799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.747},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.32700000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655392,"kb_used_data":57168,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93716448,"statfs":{"total":96636764160,"available":95965642752,"internally_reserved":0,"allocated":58540032,"data_stored":58359357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2,0,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.314}]},{"osd":1,"last update":"Sun Sep 15 03:15:20 2024","interfaces":[{"interface":"back","average":{"1min":5.133,"5min":2.7770000000000001,"15min":1.6819999999999999},"min":{"1min":0.34100000000000003,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":15.833,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.3959999999999999},{"interface":"front","average":{"1min":5.1749999999999998,"5min":2.8199999999999998,"15min":1.738},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":15.872999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":3.4249999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":7.7789999999999999},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":13.07}]}]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649484,"kb_used_data":54916,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93722356,"statfs":{"total":96636764160,"available":95971692544,"internally_reserved":0,"allocated":56233984,"data_stored":56050163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.214}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":3.996},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.38600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.255},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.0389999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":655892,"kb_used_data":57228,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93715948,"statfs":{"total":96636764160,"available":95965130752,"internally_reserved":0,"allocated":58601472,"data_stored":58417651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,13,4,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":0.38900000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.42299999999999999},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.35299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57634816,"data_stored":57634816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55508992,"data_stored":55508992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57548800,"data_stored":57548800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57622528,"data_stored":57622528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:26.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11006", 'reported_seq': 11074, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.047520+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:16:21.047520+0000', 'last_peered': '2024-09-15T03:16:21.047520+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:16:21.047520+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7405568, 'num_objects': 904, 'num_object_clones': 0, 'num_object_copies': 3616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11006, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1062}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10589", 'reported_seq': 12286, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.176291+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:16:21.176291+0000', 'last_peered': '2024-09-15T03:16:21.176291+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:16:21.176291+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6854258, 'num_objects': 837, 'num_object_clones': 0, 'num_object_copies': 3348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10587, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 816, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -349}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11304", 'reported_seq': 11414, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.282854+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:16:21.282854+0000', 'last_peered': '2024-09-15T03:16:21.282854+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:16:21.282854+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7502673, 'num_objects': 917, 'num_object_clones': 0, 'num_object_copies': 3668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 917, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11303, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1174}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11198", 'reported_seq': 11306, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.319748+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:16:21.319748+0000', 'last_peered': '2024-09-15T03:16:21.319748+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:16:21.319748+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7231090, 'num_objects': 883, 'num_object_clones': 0, 'num_object_copies': 3532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11196, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:16:26.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11150", 'reported_seq': 11236, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.120596+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:16:21.120596+0000', 'last_peered': '2024-09-15T03:16:21.120596+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:16:21.120596+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6946816, 'num_objects': 848, 'num_object_clones': 0, 'num_object_copies': 3392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11150, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1190}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11104", 'reported_seq': 11226, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.206253+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:16:21.206253+0000', 'last_peered': '2024-09-15T03:16:21.206253+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:16:21.206253+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6993522, 'num_objects': 854, 'num_object_clones': 0, 'num_object_copies': 3416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 854, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11102, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1172}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11224", 'reported_seq': 11343, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.021091+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:16:21.021091+0000', 'last_peered': '2024-09-15T03:16:21.021091+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:16:21.021091+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7050866, 'num_objects': 861, 'num_object_clones': 0, 'num_object_copies': 3444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11222, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1195}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11053", 'reported_seq': 11146, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.216742+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:16:21.216742+0000', 'last_peered': '2024-09-15T03:16:21.216742+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:16:21.216742+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7507172, 'num_objects': 917, 'num_object_clones': 0, 'num_object_copies': 3668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 917, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11050, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1110}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11059", 'reported_seq': 11180, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.277172+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:16:21.277172+0000', 'last_peered': '2024-09-15T03:16:21.277172+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:16:21.277172+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7444082, 'num_objects': 909, 'num_object_clones': 0, 'num_object_copies': 3636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11057, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1133}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:16:26.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11000", 'reported_seq': 11096, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.181714+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:16:21.181714+0000', 'last_peered': '2024-09-15T03:16:21.181714+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:16:21.181714+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7229867, 'num_objects': 883, 'num_object_clones': 0, 'num_object_copies': 3532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10997, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1179}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:26.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:16:26.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10831", 'reported_seq': 10933, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.257711+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:16:21.257711+0000', 'last_peered': '2024-09-15T03:16:21.257711+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:16:21.257711+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6968946, 'num_objects': 851, 'num_object_clones': 0, 'num_object_copies': 3404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 851, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10829, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:26.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:16:26.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'10996", 'reported_seq': 11105, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.205060+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:16:21.205060+0000', 'last_peered': '2024-09-15T03:16:21.205060+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:16:21.205060+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7263858, 'num_objects': 887, 'num_object_clones': 0, 'num_object_copies': 3548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10994, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1090}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:26.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:16:26.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11130", 'reported_seq': 11242, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:21.217191+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:16:21.217191+0000', 'last_peered': '2024-09-15T03:16:21.217191+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:16:21.217191+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7198322, 'num_objects': 879, 'num_object_clones': 0, 'num_object_copies': 3516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11128, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1180}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:26.598 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:16:28.362 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:28.363 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25601/53104 objects degraded (48.209%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":711,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13276,"data_bytes":108714580,"bytes_used":2663755776,"bytes_avail":383883300864,"bytes_total":386547056640,"degraded_objects":25601,"degraded_total":53104,"degraded_ratio":0.48209174450135583,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:16:28.467 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:16:28.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:30.072 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:30.108 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:30.131 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-09-15T03:16:28.955995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108714580,"num_objects":13276,"num_object_clones":0,"num_object_copies":53104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177431,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7083,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160658,"ondisk_log_size":160658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2601324,"kb_used_data":215308,"kb_used_omap":14,"kb_used_meta":2385969,"kb_avail":374886036,"statfs":{"total":386547056640,"available":383883300864,"internally_reserved":0,"allocated":220475392,"data_stored":219743254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2443232283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,6,6,10,10,7,7,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14557184,"num_objects":-1777,"num_object_clones":0,"num_object_copies":-7108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1777,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003068"},"pg_stats":[{"pgid":"1.f","version":"37'11057","reported_seq":11125,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.222562+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:26.222562+0000","last_peered":"2024-09-15T03:16:26.222562+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:26.222562+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6987776,"num_objects":853,"num_object_clones":0,"num_object_copies":3412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11057,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10636","reported_seq":12427,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.389247+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:26.389247+0000","last_peered":"2024-09-15T03:16:26.389247+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:26.389247+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6469234,"num_objects":790,"num_object_clones":0,"num_object_copies":3160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1139,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10634,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":863,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11348","reported_seq":11458,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.434328+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:26.434328+0000","last_peered":"2024-09-15T03:16:26.434328+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:26.434328+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7142225,"num_objects":873,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11347,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1218}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10833","reported_seq":14807,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:24.541634+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:24.541634+0000","last_peered":"2024-09-15T03:16:24.541634+0000","last_clean":"2024-09-15T03:16:24.541634+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:24.541634+0000","last_undegraded":"2024-09-15T03:16:24.541634+0000","last_fullsized":"2024-09-15T03:16:24.541634+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6510194,"num_objects":795,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10831,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11243","reported_seq":11351,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.229563+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:26.229563+0000","last_peered":"2024-09-15T03:16:26.229563+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:26.229563+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6862450,"num_objects":838,"num_object_clones":0,"num_object_copies":3352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11203","reported_seq":11288,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.332802+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:26.332802+0000","last_peered":"2024-09-15T03:16:26.332802+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:26.332802+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6512640,"num_objects":795,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1243}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11141","reported_seq":11264,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.115298+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:26.115298+0000","last_peered":"2024-09-15T03:16:26.115298+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:26.115298+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6690418,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11139,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11273","reported_seq":11392,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.268595+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:26.268595+0000","last_peered":"2024-09-15T03:16:26.268595+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:26.268595+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6649458,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11271,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1244}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11112","reported_seq":11205,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.055348+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:26.055348+0000","last_peered":"2024-09-15T03:16:26.055348+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:26.055348+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7023844,"num_objects":858,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11109,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1169}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11123","reported_seq":11244,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.346692+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:26.346692+0000","last_peered":"2024-09-15T03:16:26.346692+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:26.346692+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6919794,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11121,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1197}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11059","reported_seq":11155,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.299182+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:26.299182+0000","last_peered":"2024-09-15T03:16:26.299182+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:26.299182+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6746539,"num_objects":824,"num_object_clones":0,"num_object_copies":3296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11056,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1238}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11052","reported_seq":15095,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:24.561253+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:24.561253+0000","last_peered":"2024-09-15T03:16:24.561253+0000","last_clean":"2024-09-15T03:16:24.561253+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:24.561253+0000","last_undegraded":"2024-09-15T03:16:24.561253+0000","last_fullsized":"2024-09-15T03:16:24.561253+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6937401,"num_objects":847,"num_object_clones":0,"num_object_copies":3388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11051,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11237","reported_seq":15413,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:24.356492+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:24.356492+0000","last_peered":"2024-09-15T03:16:24.356492+0000","last_clean":"2024-09-15T03:16:24.356492+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:24.356492+0000","last_undegraded":"2024-09-15T03:16:24.356492+0000","last_fullsized":"2024-09-15T03:16:24.356492+0000","mapping_epoch":36,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7338809,"num_objects":896,"num_object_clones":0,"num_object_copies":3584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11236,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10885","reported_seq":10987,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.344623+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:26.344623+0000","last_peered":"2024-09-15T03:16:26.344623+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:26.344623+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6526578,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10883,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1216}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11055","reported_seq":11164,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.332911+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:26.332911+0000","last_peered":"2024-09-15T03:16:26.332911+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:26.332911+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6780530,"num_objects":828,"num_object_clones":0,"num_object_copies":3312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11053,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11201","reported_seq":11315,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:26.292356+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:26.292356+0000","last_peered":"2024-09-15T03:16:26.292356+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:26.292356+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6616690,"num_objects":808,"num_object_clones":0,"num_object_copies":3232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108714580,"num_objects":13276,"num_object_clones":0,"num_object_copies":53104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177431,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7083,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217866240,"data_stored":217866240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160658,"ondisk_log_size":160658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652220,"kb_used_data":53492,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93719620,"statfs":{"total":96636764160,"available":95968890880,"internally_reserved":0,"allocated":54775808,"data_stored":54591987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,0,4,5,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.21099999999999999},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.39900000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:15:25 2024","interfaces":[{"interface":"back","average":{"1min":4.5990000000000002,"5min":3.7549999999999999,"15min":2.4180000000000001},"min":{"1min":0.20699999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":35.402000000000001,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.3500000000000001,"5min":2.0779999999999998,"15min":1.6359999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":4.7569999999999997,"5min":22.262,"15min":22.262},"last":0.70199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.371},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.27100000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":656112,"kb_used_data":53736,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93715728,"statfs":{"total":96636764160,"available":95964905472,"internally_reserved":0,"allocated":55025664,"data_stored":54844989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":10.680999999999999},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.35899999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.33500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":2.7080000000000002},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":2.6840000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649484,"kb_used_data":54916,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93722356,"statfs":{"total":96636764160,"available":95971692544,"internally_reserved":0,"allocated":56233984,"data_stored":56050163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.214}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":3.996},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.38600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.255},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.0389999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643508,"kb_used_data":53164,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93728332,"statfs":{"total":96636764160,"available":95977811968,"internally_reserved":0,"allocated":54439936,"data_stored":54256115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,1,2,2,2,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:15:28 2024","interfaces":[{"interface":"back","average":{"1min":3.6600000000000001,"5min":2.2509999999999999,"15min":1.4710000000000001},"min":{"1min":0.26300000000000001,"5min":0.19400000000000001,"15min":0.17499999999999999},"max":{"1min":10.191000000000001,"5min":10.843,"15min":19.527000000000001},"last":2.9580000000000002},{"interface":"front","average":{"1min":4.3979999999999997,"5min":3.0680000000000001,"15min":2.3410000000000002},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":10.818,"5min":29.763000000000002,"15min":30.266999999999999},"last":3.2000000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":3.246},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.30499999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54124544,"data_stored":54124544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55508992,"data_stored":55508992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54083584,"data_stored":54083584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54149120,"data_stored":54149120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:33.134 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:34.524 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:34.540 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:34.554 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-15T03:16:32.960470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101620308,"num_objects":12410,"num_object_clones":0,"num_object_copies":49640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12410,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178297,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7150,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161024,"ondisk_log_size":161024,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2571116,"kb_used_data":201740,"kb_used_omap":14,"kb_used_meta":2369329,"kb_avail":374916244,"statfs":{"total":386547056640,"available":383914233856,"internally_reserved":0,"allocated":206581760,"data_stored":205849622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426192923},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,4,3,9,9,7,15,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20316160,"num_objects":-2480,"num_object_clones":0,"num_object_copies":-9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2480,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006568"},"pg_stats":[{"pgid":"1.f","version":"37'11127","reported_seq":11193,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.404231+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:31.404231+0000","last_peered":"2024-09-15T03:16:31.404231+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:31.404231+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6414336,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11127,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10683","reported_seq":12608,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.397051+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:31.397051+0000","last_peered":"2024-09-15T03:16:31.397051+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:31.397051+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084210,"num_objects":743,"num_object_clones":0,"num_object_copies":2972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11395","reported_seq":11505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.417553+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:31.417553+0000","last_peered":"2024-09-15T03:16:31.417553+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:31.417553+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6757201,"num_objects":826,"num_object_clones":0,"num_object_copies":3304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11394,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1265}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10882","reported_seq":14856,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:29.641078+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:29.641078+0000","last_peered":"2024-09-15T03:16:29.641078+0000","last_clean":"2024-09-15T03:16:29.641078+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:29.641078+0000","last_undegraded":"2024-09-15T03:16:29.641078+0000","last_fullsized":"2024-09-15T03:16:29.641078+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6108786,"num_objects":746,"num_object_clones":0,"num_object_copies":2984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10880,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11297","reported_seq":11405,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:30.559799+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:30.559799+0000","last_peered":"2024-09-15T03:16:30.559799+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:30.559799+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6420082,"num_objects":784,"num_object_clones":0,"num_object_copies":3136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11254","reported_seq":11340,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.257082+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:31.257082+0000","last_peered":"2024-09-15T03:16:31.257082+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:31.257082+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6094848,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11198","reported_seq":11321,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.406241+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:31.406241+0000","last_peered":"2024-09-15T03:16:31.406241+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:31.406241+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6223474,"num_objects":760,"num_object_clones":0,"num_object_copies":3040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11328","reported_seq":11447,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.480987+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:31.480987+0000","last_peered":"2024-09-15T03:16:31.480987+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:31.480987+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6198898,"num_objects":757,"num_object_clones":0,"num_object_copies":3028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11173","reported_seq":11266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.258133+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:31.258133+0000","last_peered":"2024-09-15T03:16:31.258133+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:31.258133+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6524132,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11170,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11181","reported_seq":11302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.294122+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:31.294122+0000","last_peered":"2024-09-15T03:16:31.294122+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:31.294122+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6444658,"num_objects":787,"num_object_clones":0,"num_object_copies":3148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1255}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11110","reported_seq":11205,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.392515+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:31.392515+0000","last_peered":"2024-09-15T03:16:31.392515+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:31.392515+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6328747,"num_objects":773,"num_object_clones":0,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11107,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11114","reported_seq":15157,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:29.609117+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:29.609117+0000","last_peered":"2024-09-15T03:16:29.609117+0000","last_clean":"2024-09-15T03:16:29.609117+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:29.609117+0000","last_undegraded":"2024-09-15T03:16:29.609117+0000","last_fullsized":"2024-09-15T03:16:29.609117+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6429497,"num_objects":785,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11113,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11297","reported_seq":15473,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:29.605080+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:29.605080+0000","last_peered":"2024-09-15T03:16:29.605080+0000","last_clean":"2024-09-15T03:16:29.605080+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:29.605080+0000","last_undegraded":"2024-09-15T03:16:29.605080+0000","last_fullsized":"2024-09-15T03:16:29.605080+0000","mapping_epoch":36,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6847289,"num_objects":836,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11296,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10942","reported_seq":11044,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.364736+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:31.364736+0000","last_peered":"2024-09-15T03:16:31.364736+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:31.364736+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6059634,"num_objects":740,"num_object_clones":0,"num_object_copies":2960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11098","reported_seq":11207,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.445963+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:31.445963+0000","last_peered":"2024-09-15T03:16:31.445963+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:31.445963+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6428274,"num_objects":785,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1192}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11245","reported_seq":11360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.365533+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:31.365533+0000","last_peered":"2024-09-15T03:16:31.365533+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:31.365533+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6256242,"num_objects":764,"num_object_clones":0,"num_object_copies":3056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11243,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101620308,"num_objects":12410,"num_object_clones":0,"num_object_copies":49640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12410,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178297,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7150,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203714560,"data_stored":203714560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161024,"ondisk_log_size":161024,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636400,"kb_used_data":50152,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93735440,"statfs":{"total":96636764160,"available":95985090560,"internally_reserved":0,"allocated":51355648,"data_stored":51171827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,3,0,2,10,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.19500000000000001},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.33400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.439},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.373}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":13.894},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.41599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639540,"kb_used_data":49644,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93732300,"statfs":{"total":96636764160,"available":95981875200,"internally_reserved":0,"allocated":50835456,"data_stored":50654781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,1,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.246}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.27400000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650308,"kb_used_data":51580,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93721532,"statfs":{"total":96636764160,"available":95970848768,"internally_reserved":0,"allocated":52817920,"data_stored":52634099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.214}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":3.996},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.38600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.255},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.0389999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644868,"kb_used_data":50364,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93726972,"statfs":{"total":96636764160,"available":95976419328,"internally_reserved":0,"allocated":51572736,"data_stored":51388915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.308},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":5.7999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":5.5449999999999999},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.28699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50536448,"data_stored":50536448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":52039680,"data_stored":52039680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50528256,"data_stored":50528256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50610176,"data_stored":50610176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:34.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:36.219 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:36.244 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:36.253 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-15T03:16:34.960999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":100391508,"num_objects":12260,"num_object_clones":0,"num_object_copies":49040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12260,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178447,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7150,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160974,"ondisk_log_size":160974,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2555172,"kb_used_data":198276,"kb_used_omap":14,"kb_used_meta":2356849,"kb_avail":374932188,"statfs":{"total":386547056640,"available":383930560512,"internally_reserved":0,"allocated":203034624,"data_stored":202302486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2413413403},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,3,10,8,6,16,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15384576,"num_objects":-1878,"num_object_clones":0,"num_object_copies":-7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1878,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006623"},"pg_stats":[{"pgid":"1.f","version":"37'11127","reported_seq":11193,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.404231+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:31.404231+0000","last_peered":"2024-09-15T03:16:31.404231+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:31.404231+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6414336,"num_objects":783,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11127,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10683","reported_seq":12608,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.397051+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:31.397051+0000","last_peered":"2024-09-15T03:16:31.397051+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:31.397051+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6084210,"num_objects":743,"num_object_clones":0,"num_object_copies":2972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11395","reported_seq":11505,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.417553+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:31.417553+0000","last_peered":"2024-09-15T03:16:31.417553+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:31.417553+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6757201,"num_objects":826,"num_object_clones":0,"num_object_copies":3304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":826,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11394,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1265}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10931","reported_seq":14905,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:34.725774+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:34.725774+0000","last_peered":"2024-09-15T03:16:34.725774+0000","last_clean":"2024-09-15T03:16:34.725774+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:34.725774+0000","last_undegraded":"2024-09-15T03:16:34.725774+0000","last_fullsized":"2024-09-15T03:16:34.725774+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5707378,"num_objects":697,"num_object_clones":0,"num_object_copies":2788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11297","reported_seq":11405,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:30.559799+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:30.559799+0000","last_peered":"2024-09-15T03:16:30.559799+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:30.559799+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6420082,"num_objects":784,"num_object_clones":0,"num_object_copies":3136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11254","reported_seq":11340,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.257082+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:31.257082+0000","last_peered":"2024-09-15T03:16:31.257082+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:31.257082+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6094848,"num_objects":744,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11254,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11198","reported_seq":11321,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.406241+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:31.406241+0000","last_peered":"2024-09-15T03:16:31.406241+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:31.406241+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6223474,"num_objects":760,"num_object_clones":0,"num_object_copies":3040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1266}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11328","reported_seq":11447,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.480987+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:31.480987+0000","last_peered":"2024-09-15T03:16:31.480987+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:31.480987+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6198898,"num_objects":757,"num_object_clones":0,"num_object_copies":3028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11173","reported_seq":11266,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.258133+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:31.258133+0000","last_peered":"2024-09-15T03:16:31.258133+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:31.258133+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6524132,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11170,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11181","reported_seq":11302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.294122+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:31.294122+0000","last_peered":"2024-09-15T03:16:31.294122+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:31.294122+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6444658,"num_objects":787,"num_object_clones":0,"num_object_copies":3148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11179,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1255}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11110","reported_seq":11205,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.392515+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:31.392515+0000","last_peered":"2024-09-15T03:16:31.392515+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:31.392515+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6328747,"num_objects":773,"num_object_clones":0,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11107,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1289}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11170","reported_seq":15213,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:34.505386+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:34.505386+0000","last_peered":"2024-09-15T03:16:34.505386+0000","last_clean":"2024-09-15T03:16:34.505386+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:34.505386+0000","last_undegraded":"2024-09-15T03:16:34.505386+0000","last_fullsized":"2024-09-15T03:16:34.505386+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5970745,"num_objects":729,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11169,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11342","reported_seq":15518,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:34.787791+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:34.787791+0000","last_peered":"2024-09-15T03:16:34.787791+0000","last_clean":"2024-09-15T03:16:34.787791+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:34.787791+0000","last_undegraded":"2024-09-15T03:16:34.787791+0000","last_fullsized":"2024-09-15T03:16:34.787791+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6478649,"num_objects":791,"num_object_clones":0,"num_object_copies":3164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11341,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10942","reported_seq":11044,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.364736+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:31.364736+0000","last_peered":"2024-09-15T03:16:31.364736+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:31.364736+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6059634,"num_objects":740,"num_object_clones":0,"num_object_copies":2960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10940,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1273}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11098","reported_seq":11207,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.445963+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:31.445963+0000","last_peered":"2024-09-15T03:16:31.445963+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:31.445963+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6428274,"num_objects":785,"num_object_clones":0,"num_object_copies":3140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1192}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11245","reported_seq":11360,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:31.365533+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:31.365533+0000","last_peered":"2024-09-15T03:16:31.365533+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:31.365533+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6256242,"num_objects":764,"num_object_clones":0,"num_object_copies":3056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11243,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":100391508,"num_objects":12260,"num_object_clones":0,"num_object_copies":49040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25534,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12260,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178447,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7150,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200040448,"data_stored":200040448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160974,"ondisk_log_size":160974,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636400,"kb_used_data":50152,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93735440,"statfs":{"total":96636764160,"available":95985090560,"internally_reserved":0,"allocated":51355648,"data_stored":51171827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,3,0,2,10,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.19500000000000001},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.33400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.439},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.373}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":13.894},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.41599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639540,"kb_used_data":49644,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93732300,"statfs":{"total":96636764160,"available":95981875200,"internally_reserved":0,"allocated":50835456,"data_stored":50654781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,1,1,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.246}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.27400000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634364,"kb_used_data":48116,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93737476,"statfs":{"total":96636764160,"available":95987175424,"internally_reserved":0,"allocated":49270784,"data_stored":49086963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,2,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.95899999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.254}]}]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644868,"kb_used_data":50364,"kb_used_omap":4,"kb_used_meta":594491,"kb_avail":93726972,"statfs":{"total":96636764160,"available":95976419328,"internally_reserved":0,"allocated":51572736,"data_stored":51388915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608759361},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.308},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":5.7999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":5.5449999999999999},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.28699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.31900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50536448,"data_stored":50536448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":48365568,"data_stored":48365568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50528256,"data_stored":50528256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50610176,"data_stored":50610176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11127", 'reported_seq': 11193, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.404231+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:16:31.404231+0000', 'last_peered': '2024-09-15T03:16:31.404231+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:16:31.404231+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6414336, 'num_objects': 783, 'num_object_clones': 0, 'num_object_copies': 3132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11127, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10683", 'reported_seq': 12608, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.397051+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:16:31.397051+0000', 'last_peered': '2024-09-15T03:16:31.397051+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:16:31.397051+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6084210, 'num_objects': 743, 'num_object_clones': 0, 'num_object_copies': 2972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1072, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10681, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 930, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11395", 'reported_seq': 11505, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.417553+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:16:31.417553+0000', 'last_peered': '2024-09-15T03:16:31.417553+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:16:31.417553+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6757201, 'num_objects': 826, 'num_object_clones': 0, 'num_object_copies': 3304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 826, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11394, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1265}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11297", 'reported_seq': 11405, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:30.559799+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:16:30.559799+0000', 'last_peered': '2024-09-15T03:16:30.559799+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:16:30.559799+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6420082, 'num_objects': 784, 'num_object_clones': 0, 'num_object_copies': 3136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11295, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1300}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:16:36.255 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11254", 'reported_seq': 11340, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.257082+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:16:31.257082+0000', 'last_peered': '2024-09-15T03:16:31.257082+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:16:31.257082+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6094848, 'num_objects': 744, 'num_object_clones': 0, 'num_object_copies': 2976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11254, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11198", 'reported_seq': 11321, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.406241+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:16:31.406241+0000', 'last_peered': '2024-09-15T03:16:31.406241+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:16:31.406241+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6223474, 'num_objects': 760, 'num_object_clones': 0, 'num_object_copies': 3040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11196, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1266}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11328", 'reported_seq': 11447, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.480987+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:16:31.480987+0000', 'last_peered': '2024-09-15T03:16:31.480987+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:16:31.480987+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6198898, 'num_objects': 757, 'num_object_clones': 0, 'num_object_copies': 3028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11326, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11173", 'reported_seq': 11266, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.258133+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:16:31.258133+0000', 'last_peered': '2024-09-15T03:16:31.258133+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:16:31.258133+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6524132, 'num_objects': 797, 'num_object_clones': 0, 'num_object_copies': 3188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11170, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1230}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11181", 'reported_seq': 11302, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.294122+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:16:31.294122+0000', 'last_peered': '2024-09-15T03:16:31.294122+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:16:31.294122+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6444658, 'num_objects': 787, 'num_object_clones': 0, 'num_object_copies': 3148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 787, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11179, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1255}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:16:36.256 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11110", 'reported_seq': 11205, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.392515+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:16:31.392515+0000', 'last_peered': '2024-09-15T03:16:31.392515+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:16:31.392515+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6328747, 'num_objects': 773, 'num_object_clones': 0, 'num_object_copies': 3092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11107, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1289}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:16:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'10942", 'reported_seq': 11044, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.364736+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:16:31.364736+0000', 'last_peered': '2024-09-15T03:16:31.364736+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:16:31.364736+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6059634, 'num_objects': 740, 'num_object_clones': 0, 'num_object_copies': 2960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10940, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:16:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11098", 'reported_seq': 11207, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.445963+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:16:31.445963+0000', 'last_peered': '2024-09-15T03:16:31.445963+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:16:31.445963+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6428274, 'num_objects': 785, 'num_object_clones': 0, 'num_object_copies': 3140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11096, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1192}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:16:36.257 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11245", 'reported_seq': 11360, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:31.365533+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:16:31.365533+0000', 'last_peered': '2024-09-15T03:16:31.365533+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:16:31.365533+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6256242, 'num_objects': 764, 'num_object_clones': 0, 'num_object_copies': 3056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11243, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1295}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:36.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:16:38.340 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:38.340 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25478/46136 objects degraded (55.224%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":721,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11534,"data_bytes":94444116,"bytes_used":2617831424,"bytes_avail":383929225216,"bytes_total":386547056640,"degraded_objects":25478,"degraded_total":46136,"degraded_ratio":0.5522368649211028,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":202},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:16:38.439 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:16:38.439 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:40.103 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:40.174 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:40.236 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-15T03:16:38.961993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":94444116,"num_objects":11534,"num_object_clones":0,"num_object_copies":46136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11534,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179173,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7206,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2556476,"kb_used_data":187076,"kb_used_omap":14,"kb_used_meta":2369329,"kb_avail":374930884,"statfs":{"total":386547056640,"available":383929225216,"internally_reserved":0,"allocated":191565824,"data_stored":190833686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426192923},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,1,4,6,12,6,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14270464,"num_objects":-1742,"num_object_clones":0,"num_object_copies":-6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1742,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006538"},"pg_stats":[{"pgid":"1.f","version":"37'11174","reported_seq":11242,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.161801+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:36.161801+0000","last_peered":"2024-09-15T03:16:36.161801+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:36.161801+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":736,"num_object_clones":0,"num_object_copies":2944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11174,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10728","reported_seq":12765,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.386467+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:36.386467+0000","last_peered":"2024-09-15T03:16:36.386467+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:36.386467+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5715570,"num_objects":698,"num_object_clones":0,"num_object_copies":2792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10726,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11460","reported_seq":11566,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.436702+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:36.436702+0000","last_peered":"2024-09-15T03:16:36.436702+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:36.436702+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6224721,"num_objects":761,"num_object_clones":0,"num_object_copies":3044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11459,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1330}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10931","reported_seq":14905,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:34.725774+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:34.725774+0000","last_peered":"2024-09-15T03:16:34.725774+0000","last_clean":"2024-09-15T03:16:34.725774+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:34.725774+0000","last_undegraded":"2024-09-15T03:16:34.725774+0000","last_fullsized":"2024-09-15T03:16:34.725774+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5707378,"num_objects":697,"num_object_clones":0,"num_object_copies":2788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10929,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11362","reported_seq":11470,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.275921+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:36.275921+0000","last_peered":"2024-09-15T03:16:36.275921+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:36.275921+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5887602,"num_objects":719,"num_object_clones":0,"num_object_copies":2876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11360,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11309","reported_seq":11395,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.316507+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:36.316507+0000","last_peered":"2024-09-15T03:16:36.316507+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:36.316507+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5644288,"num_objects":689,"num_object_clones":0,"num_object_copies":2756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11309,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11247","reported_seq":11370,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.317853+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:36.317853+0000","last_peered":"2024-09-15T03:16:36.317853+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:36.317853+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5822066,"num_objects":711,"num_object_clones":0,"num_object_copies":2844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11245,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1315}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11389","reported_seq":11508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.480113+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:36.480113+0000","last_peered":"2024-09-15T03:16:36.480113+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:36.480113+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5699186,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11387,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1360}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11222","reported_seq":11315,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.108424+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:36.108424+0000","last_peered":"2024-09-15T03:16:36.108424+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:36.108424+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6122724,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11235","reported_seq":11356,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.374903+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:36.374903+0000","last_peered":"2024-09-15T03:16:36.374903+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:36.374903+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6002290,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1309}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11161","reported_seq":11255,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.320481+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:36.320481+0000","last_peered":"2024-09-15T03:16:36.320481+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:36.320481+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5910955,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11170","reported_seq":15213,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:34.505386+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:34.505386+0000","last_peered":"2024-09-15T03:16:34.505386+0000","last_clean":"2024-09-15T03:16:34.505386+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:34.505386+0000","last_undegraded":"2024-09-15T03:16:34.505386+0000","last_fullsized":"2024-09-15T03:16:34.505386+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5970745,"num_objects":729,"num_object_clones":0,"num_object_copies":2916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11169,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11342","reported_seq":15518,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:34.787791+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:34.787791+0000","last_peered":"2024-09-15T03:16:34.787791+0000","last_clean":"2024-09-15T03:16:34.787791+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:34.787791+0000","last_undegraded":"2024-09-15T03:16:34.787791+0000","last_fullsized":"2024-09-15T03:16:34.787791+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6478649,"num_objects":791,"num_object_clones":0,"num_object_copies":3164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11341,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'10995","reported_seq":11097,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.279092+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:36.279092+0000","last_peered":"2024-09-15T03:16:36.279092+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:36.279092+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5625458,"num_objects":687,"num_object_clones":0,"num_object_copies":2748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10993,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1326}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11179","reported_seq":11287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.418125+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:36.418125+0000","last_peered":"2024-09-15T03:16:36.418125+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:36.418125+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764722,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11177,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1273}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11296","reported_seq":11411,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:36.548834+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:36.548834+0000","last_peered":"2024-09-15T03:16:36.548834+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:36.548834+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5838450,"num_objects":713,"num_object_clones":0,"num_object_copies":2852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":94444116,"num_objects":11534,"num_object_clones":0,"num_object_copies":46136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25478,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11534,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179173,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7206,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":189251584,"data_stored":189251584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636984,"kb_used_data":46576,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93734856,"statfs":{"total":96636764160,"available":95984492544,"internally_reserved":0,"allocated":47693824,"data_stored":47510003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,6,3,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.19500000000000001},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.33400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":2.1720000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":1.9850000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640292,"kb_used_data":46212,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93731548,"statfs":{"total":96636764160,"available":95981105152,"internally_reserved":0,"allocated":47321088,"data_stored":47140413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":4.5019999999999998},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.8900000000000001},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":3.9260000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.36599999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":634364,"kb_used_data":48116,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93737476,"statfs":{"total":96636764160,"available":95987175424,"internally_reserved":0,"allocated":49270784,"data_stored":49086963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,2,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.38600000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.95899999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.254}]}]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644836,"kb_used_data":46172,"kb_used_omap":4,"kb_used_meta":598651,"kb_avail":93727004,"statfs":{"total":96636764160,"available":95976452096,"internally_reserved":0,"allocated":47280128,"data_stored":47096307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613019201},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.51800000000000002},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.32700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.45700000000000002},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.30599999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.36299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46915584,"data_stored":46915584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":48365568,"data_stored":48365568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46964736,"data_stored":46964736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":47005696,"data_stored":47005696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:43.238 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:45.169 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:45.237 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:45.266 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-15T03:16:44.963478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":87349844,"num_objects":10668,"num_object_clones":0,"num_object_copies":42672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10668,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180039,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7277,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2543228,"kb_used_data":173828,"kb_used_omap":14,"kb_used_meta":2369329,"kb_avail":374944132,"statfs":{"total":386547056640,"available":383942791168,"internally_reserved":0,"allocated":177999872,"data_stored":177267734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2426192923},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,7,9,6,13,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14270464,"num_objects":-1742,"num_object_clones":0,"num_object_copies":-6968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1742,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003008"},"pg_stats":[{"pgid":"1.f","version":"37'11237","reported_seq":11302,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.249495+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:41.249495+0000","last_peered":"2024-09-15T03:16:41.249495+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:41.249495+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5513216,"num_objects":673,"num_object_clones":0,"num_object_copies":2692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11237,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1293}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10773","reported_seq":12952,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.553433+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:41.553433+0000","last_peered":"2024-09-15T03:16:41.553433+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:41.553433+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5346930,"num_objects":653,"num_object_clones":0,"num_object_copies":2612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":945,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1057,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11522","reported_seq":11632,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.481062+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:41.481062+0000","last_peered":"2024-09-15T03:16:41.481062+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:41.481062+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5716817,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11521,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1392}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'10988","reported_seq":14960,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:39.824143+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:39.824143+0000","last_peered":"2024-09-15T03:16:39.824143+0000","last_clean":"2024-09-15T03:16:39.824143+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:39.824143+0000","last_undegraded":"2024-09-15T03:16:39.824143+0000","last_fullsized":"2024-09-15T03:16:39.824143+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5240434,"num_objects":640,"num_object_clones":0,"num_object_copies":2560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10986,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11412","reported_seq":11520,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.404140+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:41.404140+0000","last_peered":"2024-09-15T03:16:41.404140+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:41.404140+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5478002,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11410,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11357","reported_seq":11443,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.084019+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:41.084019+0000","last_peered":"2024-09-15T03:16:41.084019+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:41.084019+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5251072,"num_objects":641,"num_object_clones":0,"num_object_copies":2564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11357,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1397}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11299","reported_seq":11422,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.522419+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:41.522419+0000","last_peered":"2024-09-15T03:16:41.522419+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:41.522419+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5396082,"num_objects":659,"num_object_clones":0,"num_object_copies":2636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11297,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1367}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11452","reported_seq":11570,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.544270+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:41.544270+0000","last_peered":"2024-09-15T03:16:41.544270+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:41.544270+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5183090,"num_objects":633,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11278","reported_seq":11371,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.430294+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:41.430294+0000","last_peered":"2024-09-15T03:16:41.430294+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:41.430294+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5663972,"num_objects":692,"num_object_clones":0,"num_object_copies":2768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11275,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11289","reported_seq":11410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.323689+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:41.323689+0000","last_peered":"2024-09-15T03:16:41.323689+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:41.323689+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5559922,"num_objects":679,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1363}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11209","reported_seq":11304,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.418846+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:41.418846+0000","last_peered":"2024-09-15T03:16:41.418846+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:41.418846+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5517739,"num_objects":674,"num_object_clones":0,"num_object_copies":2696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11206,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1388}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11231","reported_seq":15274,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:39.934966+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:39.934966+0000","last_peered":"2024-09-15T03:16:39.934966+0000","last_clean":"2024-09-15T03:16:39.934966+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:39.934966+0000","last_undegraded":"2024-09-15T03:16:39.934966+0000","last_fullsized":"2024-09-15T03:16:39.934966+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5471033,"num_objects":668,"num_object_clones":0,"num_object_copies":2672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11230,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11400","reported_seq":15575,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:39.836789+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:39.836789+0000","last_peered":"2024-09-15T03:16:39.836789+0000","last_clean":"2024-09-15T03:16:39.836789+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:39.836789+0000","last_undegraded":"2024-09-15T03:16:39.836789+0000","last_fullsized":"2024-09-15T03:16:39.836789+0000","mapping_epoch":36,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6003513,"num_objects":733,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11399,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11037","reported_seq":11139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.183142+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:41.183142+0000","last_peered":"2024-09-15T03:16:41.183142+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:41.183142+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5281394,"num_objects":645,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11035,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1368}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11235","reported_seq":11344,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.466183+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:41.466183+0000","last_peered":"2024-09-15T03:16:41.466183+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:41.466183+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5305970,"num_objects":648,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11233,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11347","reported_seq":11462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:41.493823+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:41.493823+0000","last_peered":"2024-09-15T03:16:41.493823+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:41.493823+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5420658,"num_objects":662,"num_object_clones":0,"num_object_copies":2648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11345,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":87349844,"num_objects":10668,"num_object_clones":0,"num_object_copies":42672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10668,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180039,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7277,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174972928,"data_stored":174972928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":637696,"kb_used_data":43128,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93734144,"statfs":{"total":96636764160,"available":95983763456,"internally_reserved":0,"allocated":44163072,"data_stored":43979251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,4,2,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":0.49299999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.314}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.372},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.24399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640956,"kb_used_data":42732,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93730884,"statfs":{"total":96636764160,"available":95980425216,"internally_reserved":0,"allocated":43757568,"data_stored":43576893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":1.5580000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":1.536},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":1.7669999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":1.7190000000000001},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":1.603}]}]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":635084,"kb_used_data":44660,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93736756,"statfs":{"total":96636764160,"available":95986438144,"internally_reserved":0,"allocated":45731840,"data_stored":45548019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,3,1,7,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":4.976}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.38400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.44}]}]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629492,"kb_used_data":43308,"kb_used_omap":4,"kb_used_meta":586171,"kb_avail":93742348,"statfs":{"total":96636764160,"available":95992164352,"internally_reserved":0,"allocated":44347392,"data_stored":44163571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600239681},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.26900000000000002},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.61599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.51700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.55700000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43376640,"data_stored":43376640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":44720128,"data_stored":44720128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":43429888,"data_stored":43429888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":43446272,"data_stored":43446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:45.267 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:47.413 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:47.438 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:47.486 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-15T03:16:46.963937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80124500,"num_objects":9786,"num_object_clones":0,"num_object_copies":39144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9786,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180921,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7324,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160948,"ondisk_log_size":160948,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2531896,"kb_used_data":158344,"kb_used_omap":14,"kb_used_meta":2373489,"kb_avail":374955464,"statfs":{"total":386547056640,"available":383954395136,"internally_reserved":0,"allocated":162144256,"data_stored":161412118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2430452763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,5,6,10,14,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20267008,"num_objects":-2474,"num_object_clones":0,"num_object_copies":-9896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"1.f","version":"37'11288","reported_seq":11356,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.225607+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:46.225607+0000","last_peered":"2024-09-15T03:16:46.225607+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:46.225607+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5095424,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10829","reported_seq":13102,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.567721+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:46.567721+0000","last_peered":"2024-09-15T03:16:46.567721+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:46.567721+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4888178,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11594","reported_seq":11704,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.454701+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:46.454701+0000","last_peered":"2024-09-15T03:16:46.454701+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:46.454701+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126993,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11593,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11040","reported_seq":15014,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:44.968217+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:44.968217+0000","last_peered":"2024-09-15T03:16:44.968217+0000","last_clean":"2024-09-15T03:16:44.968217+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:44.968217+0000","last_undegraded":"2024-09-15T03:16:44.968217+0000","last_fullsized":"2024-09-15T03:16:44.968217+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4814450,"num_objects":588,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11038,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11462","reported_seq":11569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.325437+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:46.325437+0000","last_peered":"2024-09-15T03:16:46.325437+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:46.325437+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5068402,"num_objects":619,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11400","reported_seq":11486,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.432834+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:46.432834+0000","last_peered":"2024-09-15T03:16:46.432834+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:46.432834+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4898816,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11400,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11352","reported_seq":11475,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.535462+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:46.535462+0000","last_peered":"2024-09-15T03:16:46.535462+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:46.535462+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4961906,"num_objects":606,"num_object_clones":0,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11350,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11508","reported_seq":11627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.433785+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:46.433785+0000","last_peered":"2024-09-15T03:16:46.433785+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:46.433785+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4724338,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1479}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11336","reported_seq":11428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.574509+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:46.574509+0000","last_peered":"2024-09-15T03:16:46.574509+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:46.574509+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5188836,"num_objects":634,"num_object_clones":0,"num_object_copies":2536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11348","reported_seq":11469,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.476693+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:46.476693+0000","last_peered":"2024-09-15T03:16:46.476693+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:46.476693+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5076594,"num_objects":620,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1422}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11267","reported_seq":11363,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.333281+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:46.333281+0000","last_peered":"2024-09-15T03:16:46.333281+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:46.333281+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5042603,"num_objects":616,"num_object_clones":0,"num_object_copies":2464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11264,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11284","reported_seq":15327,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:45.142921+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:45.142921+0000","last_peered":"2024-09-15T03:16:45.142921+0000","last_clean":"2024-09-15T03:16:45.142921+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:45.142921+0000","last_undegraded":"2024-09-15T03:16:45.142921+0000","last_fullsized":"2024-09-15T03:16:45.142921+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5036857,"num_objects":615,"num_object_clones":0,"num_object_copies":2460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11283,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11466","reported_seq":15642,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:44.788186+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:44.788186+0000","last_peered":"2024-09-15T03:16:44.788186+0000","last_clean":"2024-09-15T03:16:44.788186+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:44.788186+0000","last_undegraded":"2024-09-15T03:16:44.788186+0000","last_fullsized":"2024-09-15T03:16:44.788186+0000","mapping_epoch":36,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462841,"num_objects":667,"num_object_clones":0,"num_object_copies":2668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11465,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11085","reported_seq":11186,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.529158+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:46.529158+0000","last_peered":"2024-09-15T03:16:46.529158+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:46.529158+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4888178,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1416}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11283","reported_seq":11392,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.345488+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:46.345488+0000","last_peered":"2024-09-15T03:16:46.345488+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:46.345488+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4912754,"num_objects":600,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1377}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11406","reported_seq":11521,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.536749+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:46.536749+0000","last_peered":"2024-09-15T03:16:46.536749+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:46.536749+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4937330,"num_objects":603,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":80124500,"num_objects":9786,"num_object_clones":0,"num_object_copies":39144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9786,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180921,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7324,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":160415744,"data_stored":160415744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160948,"ondisk_log_size":160948,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638344,"kb_used_data":39608,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93733496,"statfs":{"total":96636764160,"available":95983099904,"internally_reserved":0,"allocated":40558592,"data_stored":40374771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":7.3869999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.35399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.505},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.45700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624944,"kb_used_data":39208,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93746896,"statfs":{"total":96636764160,"available":95996821504,"internally_reserved":0,"allocated":40148992,"data_stored":39968317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,5,3,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.35099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":5.7389999999999999},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":5.766}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":5.6929999999999996},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.39100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639116,"kb_used_data":40388,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93732724,"statfs":{"total":96636764160,"available":95982309376,"internally_reserved":0,"allocated":41357312,"data_stored":41173491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.32200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.33600000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629492,"kb_used_data":39140,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93742348,"statfs":{"total":96636764160,"available":95992164352,"internally_reserved":0,"allocated":40079360,"data_stored":39895539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.26700000000000002},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.307}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.34200000000000003},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.39300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":1.379},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39751680,"data_stored":39751680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":41033728,"data_stored":41033728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39821312,"data_stored":39821312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39809024,"data_stored":39809024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:47.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:16:47.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11288", 'reported_seq': 11356, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.225607+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:16:46.225607+0000', 'last_peered': '2024-09-15T03:16:46.225607+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:16:46.225607+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5095424, 'num_objects': 622, 'num_object_clones': 0, 'num_object_copies': 2488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11288, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1344}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:47.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:16:47.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10829", 'reported_seq': 13102, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.567721+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:16:46.567721+0000', 'last_peered': '2024-09-15T03:16:46.567721+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:16:46.567721+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4888178, 'num_objects': 597, 'num_object_clones': 0, 'num_object_copies': 2388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 898, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10827, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1104, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:47.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:16:47.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11594", 'reported_seq': 11704, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.454701+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:16:46.454701+0000', 'last_peered': '2024-09-15T03:16:46.454701+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:16:46.454701+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5126993, 'num_objects': 627, 'num_object_clones': 0, 'num_object_copies': 2508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 627, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11593, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1464}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:47.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:16:47.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11462", 'reported_seq': 11569, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.325437+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:16:46.325437+0000', 'last_peered': '2024-09-15T03:16:46.325437+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:16:46.325437+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5068402, 'num_objects': 619, 'num_object_clones': 0, 'num_object_copies': 2476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 619, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11460, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1465}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:47.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:16:47.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11400", 'reported_seq': 11486, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.432834+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:16:46.432834+0000', 'last_peered': '2024-09-15T03:16:46.432834+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:16:46.432834+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4898816, 'num_objects': 598, 'num_object_clones': 0, 'num_object_copies': 2392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11400, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1440}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:47.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:16:47.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11352", 'reported_seq': 11475, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.535462+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:16:46.535462+0000', 'last_peered': '2024-09-15T03:16:46.535462+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:16:46.535462+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4961906, 'num_objects': 606, 'num_object_clones': 0, 'num_object_copies': 2424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11350, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1420}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:47.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:16:47.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11508", 'reported_seq': 11627, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.433785+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:16:46.433785+0000', 'last_peered': '2024-09-15T03:16:46.433785+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:16:46.433785+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4724338, 'num_objects': 577, 'num_object_clones': 0, 'num_object_copies': 2308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11506, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1479}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:47.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:16:47.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11336", 'reported_seq': 11428, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.574509+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:16:46.574509+0000', 'last_peered': '2024-09-15T03:16:46.574509+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:16:46.574509+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5188836, 'num_objects': 634, 'num_object_clones': 0, 'num_object_copies': 2536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11333, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1393}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:47.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:16:47.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11348", 'reported_seq': 11469, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.476693+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:16:46.476693+0000', 'last_peered': '2024-09-15T03:16:46.476693+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:16:46.476693+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5076594, 'num_objects': 620, 'num_object_clones': 0, 'num_object_copies': 2480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11346, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1422}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:47.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:16:47.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11267", 'reported_seq': 11363, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.333281+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:16:46.333281+0000', 'last_peered': '2024-09-15T03:16:46.333281+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:16:46.333281+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5042603, 'num_objects': 616, 'num_object_clones': 0, 'num_object_copies': 2464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11264, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1446}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:47.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:16:47.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11085", 'reported_seq': 11186, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.529158+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:16:46.529158+0000', 'last_peered': '2024-09-15T03:16:46.529158+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:16:46.529158+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4888178, 'num_objects': 597, 'num_object_clones': 0, 'num_object_copies': 2388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11083, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1416}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:47.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:16:47.491 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11283", 'reported_seq': 11392, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.345488+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:16:46.345488+0000', 'last_peered': '2024-09-15T03:16:46.345488+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:16:46.345488+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4912754, 'num_objects': 600, 'num_object_clones': 0, 'num_object_copies': 2400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11281, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1377}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:47.491 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:16:47.492 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11406", 'reported_seq': 11521, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:46.536749+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:16:46.536749+0000', 'last_peered': '2024-09-15T03:16:46.536749+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:16:46.536749+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4937330, 'num_objects': 603, 'num_object_clones': 0, 'num_object_copies': 2412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11404, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1456}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:47.492 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:16:50.069 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:50.069 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25360/39144 objects degraded (64.786%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":733,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9786,"data_bytes":80124500,"bytes_used":2592661504,"bytes_avail":383954395136,"bytes_total":386547056640,"degraded_objects":25360,"degraded_total":39144,"degraded_ratio":0.64786429593296546,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":145},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:16:50.150 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:16:50.151 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:52.020 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:52.066 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:52.119 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-15T03:16:50.964926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":78625364,"num_objects":9603,"num_object_clones":0,"num_object_copies":38412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181104,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7324,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160931,"ondisk_log_size":160931,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2516036,"kb_used_data":154964,"kb_used_omap":14,"kb_used_meta":2361009,"kb_avail":374971324,"statfs":{"total":386547056640,"available":383970635776,"internally_reserved":0,"allocated":158683136,"data_stored":157950998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":2417673243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,4,9,9,17,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15818752,"num_objects":-1931,"num_object_clones":0,"num_object_copies":-7724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1931,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002932"},"pg_stats":[{"pgid":"1.f","version":"37'11288","reported_seq":11356,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.225607+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:46.225607+0000","last_peered":"2024-09-15T03:16:46.225607+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:46.225607+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5095424,"num_objects":622,"num_object_clones":0,"num_object_copies":2488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11288,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1344}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10829","reported_seq":13102,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.567721+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:46.567721+0000","last_peered":"2024-09-15T03:16:46.567721+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:46.567721+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4888178,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10827,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-301}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11594","reported_seq":11704,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.454701+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:46.454701+0000","last_peered":"2024-09-15T03:16:46.454701+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:46.454701+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126993,"num_objects":627,"num_object_clones":0,"num_object_copies":2508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11593,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1464}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11090","reported_seq":15064,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:50.216615+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:50.216615+0000","last_peered":"2024-09-15T03:16:50.216615+0000","last_clean":"2024-09-15T03:16:50.216615+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:50.216615+0000","last_undegraded":"2024-09-15T03:16:50.216615+0000","last_fullsized":"2024-09-15T03:16:50.216615+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404850,"num_objects":538,"num_object_clones":0,"num_object_copies":2152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11088,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11462","reported_seq":11569,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.325437+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:46.325437+0000","last_peered":"2024-09-15T03:16:46.325437+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:46.325437+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5068402,"num_objects":619,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11400","reported_seq":11486,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.432834+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:46.432834+0000","last_peered":"2024-09-15T03:16:46.432834+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:46.432834+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4898816,"num_objects":598,"num_object_clones":0,"num_object_copies":2392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11400,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1440}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11352","reported_seq":11475,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.535462+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:46.535462+0000","last_peered":"2024-09-15T03:16:46.535462+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:46.535462+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4961906,"num_objects":606,"num_object_clones":0,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11350,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11508","reported_seq":11627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.433785+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:46.433785+0000","last_peered":"2024-09-15T03:16:46.433785+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:46.433785+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4724338,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1479}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11336","reported_seq":11428,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.574509+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:46.574509+0000","last_peered":"2024-09-15T03:16:46.574509+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:46.574509+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5188836,"num_objects":634,"num_object_clones":0,"num_object_copies":2536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11333,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11348","reported_seq":11469,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.476693+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:46.476693+0000","last_peered":"2024-09-15T03:16:46.476693+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:46.476693+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5076594,"num_objects":620,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1422}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11267","reported_seq":11363,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.333281+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:46.333281+0000","last_peered":"2024-09-15T03:16:46.333281+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:46.333281+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5042603,"num_objects":616,"num_object_clones":0,"num_object_copies":2464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11264,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11344","reported_seq":15386,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:50.252961+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:50.252961+0000","last_peered":"2024-09-15T03:16:50.252961+0000","last_clean":"2024-09-15T03:16:50.252961+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:50.252961+0000","last_undegraded":"2024-09-15T03:16:50.252961+0000","last_fullsized":"2024-09-15T03:16:50.252961+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4545337,"num_objects":555,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11343,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11539","reported_seq":15715,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:50.155566+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:50.155566+0000","last_peered":"2024-09-15T03:16:50.155566+0000","last_clean":"2024-09-15T03:16:50.155566+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:50.155566+0000","last_undegraded":"2024-09-15T03:16:50.155566+0000","last_fullsized":"2024-09-15T03:16:50.155566+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4864825,"num_objects":594,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11538,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11085","reported_seq":11186,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.529158+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:46.529158+0000","last_peered":"2024-09-15T03:16:46.529158+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:46.529158+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4888178,"num_objects":597,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11083,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1416}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11283","reported_seq":11392,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.345488+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:46.345488+0000","last_peered":"2024-09-15T03:16:46.345488+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:46.345488+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4912754,"num_objects":600,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1377}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11406","reported_seq":11521,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:46.536749+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:46.536749+0000","last_peered":"2024-09-15T03:16:46.536749+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:46.536749+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4937330,"num_objects":603,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1456}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":78625364,"num_objects":9603,"num_object_clones":0,"num_object_copies":38412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181104,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7324,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":156860416,"data_stored":156860416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160931,"ondisk_log_size":160931,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":638344,"kb_used_data":39608,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93733496,"statfs":{"total":96636764160,"available":95983099904,"internally_reserved":0,"allocated":40558592,"data_stored":40374771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.546,"15min":1.7090000000000001},"min":{"1min":0.19400000000000001,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":16.449000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.7400000000000002,"5min":2.1949999999999998,"15min":2.0910000000000002},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":8.8279999999999994,"5min":26.143000000000001,"15min":26.143000000000001},"last":7.3869999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.35399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:15:47 2024","interfaces":[{"interface":"back","average":{"1min":3.875,"5min":4.0940000000000003,"15min":2.2970000000000002},"min":{"1min":0.29999999999999999,"5min":0.23100000000000001,"15min":0.20200000000000001},"max":{"1min":12.426,"5min":40.784999999999997,"15min":40.784999999999997},"last":0.505},{"interface":"front","average":{"1min":1.111,"5min":2.2690000000000001,"15min":1.673},"min":{"1min":0.28100000000000003,"5min":0.22800000000000001,"15min":0.188},"max":{"1min":8.7769999999999992,"5min":32.377000000000002,"15min":32.377000000000002},"last":0.45700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624944,"kb_used_data":39208,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93746896,"statfs":{"total":96636764160,"available":95996821504,"internally_reserved":0,"allocated":40148992,"data_stored":39968317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,5,3,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":4.6289999999999996,"5min":3.1299999999999999,"15min":1.998},"min":{"1min":0.30599999999999999,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":19.388000000000002,"5min":31.175000000000001,"15min":31.175000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":1.9450000000000001,"5min":1.659,"15min":1.423},"min":{"1min":0.20200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":19.189,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.35099999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":5.7389999999999999},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":5.766}]},{"osd":3,"last update":"Sun Sep 15 03:15:43 2024","interfaces":[{"interface":"back","average":{"1min":3.3109999999999999,"5min":2.3740000000000001,"15min":1.6619999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":18.731000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":5.6929999999999996},{"interface":"front","average":{"1min":3.403,"5min":2.4169999999999998,"15min":1.76},"min":{"1min":0.255,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":22.367999999999999,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.39100000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623256,"kb_used_data":37008,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93748584,"statfs":{"total":96636764160,"available":95998550016,"internally_reserved":0,"allocated":37896192,"data_stored":37712371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":12.882999999999999},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":12.271000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.21},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":9.2989999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.2409999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629492,"kb_used_data":39140,"kb_used_omap":4,"kb_used_meta":590331,"kb_avail":93742348,"statfs":{"total":96636764160,"available":95992164352,"internally_reserved":0,"allocated":40079360,"data_stored":39895539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604499521},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,0,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.26700000000000002},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.307}]},{"osd":2,"last update":"Sun Sep 15 03:15:46 2024","interfaces":[{"interface":"back","average":{"1min":2.6389999999999998,"5min":2.5630000000000002,"15min":2.0590000000000002},"min":{"1min":0.221,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":10.879,"5min":20.119,"15min":30.802},"last":0.34200000000000003},{"interface":"front","average":{"1min":3.2629999999999999,"5min":2.411,"15min":1.538},"min":{"1min":0.23100000000000001,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":12.281000000000001,"5min":20.387,"15min":20.387},"last":0.39300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":1.379},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39751680,"data_stored":39751680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37478400,"data_stored":37478400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":39821312,"data_stored":39821312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":39809024,"data_stored":39809024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:55.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:56.829 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:56.833 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:56.893 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-15T03:16:54.965932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":73112148,"num_objects":8930,"num_object_clones":0,"num_object_copies":35720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8930,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181777,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7373,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3488556,"kb_used_data":145012,"kb_used_omap":14,"kb_used_meta":3343473,"kb_avail":373998804,"statfs":{"total":386547056640,"available":382974775296,"internally_reserved":0,"allocated":148492288,"data_stored":147760150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3423716379},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,9,4,12,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14237696,"num_objects":-1738,"num_object_clones":0,"num_object_copies":-6952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1738,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002965"},"pg_stats":[{"pgid":"1.f","version":"37'11342","reported_seq":11408,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.153950+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:51.153950+0000","last_peered":"2024-09-15T03:16:51.153950+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:51.153950+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":568,"num_object_clones":0,"num_object_copies":2272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11342,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10888","reported_seq":13259,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.718392+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:51.718392+0000","last_peered":"2024-09-15T03:16:51.718392+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:51.718392+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404850,"num_objects":538,"num_object_clones":0,"num_object_copies":2152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10886,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11645","reported_seq":11755,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.552471+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:51.552471+0000","last_peered":"2024-09-15T03:16:51.552471+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:51.552471+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4709201,"num_objects":576,"num_object_clones":0,"num_object_copies":2304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":576,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11644,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11090","reported_seq":15064,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:50.216615+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:50.216615+0000","last_peered":"2024-09-15T03:16:50.216615+0000","last_clean":"2024-09-15T03:16:50.216615+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:50.216615+0000","last_undegraded":"2024-09-15T03:16:50.216615+0000","last_fullsized":"2024-09-15T03:16:50.216615+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404850,"num_objects":538,"num_object_clones":0,"num_object_copies":2152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11088,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11511","reported_seq":11619,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.505645+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:51.505645+0000","last_peered":"2024-09-15T03:16:51.505645+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:51.505645+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4666994,"num_objects":570,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1514}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11445","reported_seq":11531,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.523184+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:51.523184+0000","last_peered":"2024-09-15T03:16:51.523184+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:51.523184+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4530176,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11445,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11414","reported_seq":11537,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.694897+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:51.694897+0000","last_peered":"2024-09-15T03:16:51.694897+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:51.694897+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4454002,"num_objects":544,"num_object_clones":0,"num_object_copies":2176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11412,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1482}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11555","reported_seq":11674,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.733851+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:51.733851+0000","last_peered":"2024-09-15T03:16:51.733851+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:51.733851+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4339314,"num_objects":530,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11553,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11398","reported_seq":11491,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.620419+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:51.620419+0000","last_peered":"2024-09-15T03:16:51.620419+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:51.620419+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4680932,"num_objects":572,"num_object_clones":0,"num_object_copies":2288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11395,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11401","reported_seq":11522,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.470155+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:51.470155+0000","last_peered":"2024-09-15T03:16:51.470155+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:51.470155+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4642418,"num_objects":567,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11399,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1475}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11318","reported_seq":11414,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.552800+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:51.552800+0000","last_peered":"2024-09-15T03:16:51.552800+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:51.552800+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4624811,"num_objects":565,"num_object_clones":0,"num_object_copies":2260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11315,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11344","reported_seq":15386,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:50.252961+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:50.252961+0000","last_peered":"2024-09-15T03:16:50.252961+0000","last_clean":"2024-09-15T03:16:50.252961+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:50.252961+0000","last_undegraded":"2024-09-15T03:16:50.252961+0000","last_fullsized":"2024-09-15T03:16:50.252961+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4545337,"num_objects":555,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11343,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11539","reported_seq":15715,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:50.155566+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:50.155566+0000","last_peered":"2024-09-15T03:16:50.155566+0000","last_clean":"2024-09-15T03:16:50.155566+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:50.155566+0000","last_undegraded":"2024-09-15T03:16:50.155566+0000","last_fullsized":"2024-09-15T03:16:50.155566+0000","mapping_epoch":36,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4864825,"num_objects":594,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11538,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11128","reported_seq":11230,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.505352+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:51.505352+0000","last_peered":"2024-09-15T03:16:51.505352+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:51.505352+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4535922,"num_objects":554,"num_object_clones":0,"num_object_copies":2216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1459}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11330","reported_seq":11439,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.359895+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:51.359895+0000","last_peered":"2024-09-15T03:16:51.359895+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:51.359895+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4527730,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11328,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1424}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11456","reported_seq":11571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:51.641060+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:51.641060+0000","last_peered":"2024-09-15T03:16:51.641060+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:51.641060+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4527730,"num_objects":553,"num_object_clones":0,"num_object_copies":2212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1506}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":73112148,"num_objects":8930,"num_object_clones":0,"num_object_copies":35720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8930,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181777,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7373,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":146509824,"data_stored":146509824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951336,"kb_used_data":36256,"kb_used_omap":4,"kb_used_meta":915067,"kb_avail":93420504,"statfs":{"total":96636764160,"available":95662596096,"internally_reserved":0,"allocated":37126144,"data_stored":36942323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937029185},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.40300000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":15.096},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":14.978999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.38200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":954788,"kb_used_data":35996,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93417052,"statfs":{"total":96636764160,"available":95659061248,"internally_reserved":0,"allocated":36859904,"data_stored":36679229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,4,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":20.701000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":5.8419999999999996},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":5.8879999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623256,"kb_used_data":37008,"kb_used_omap":4,"kb_used_meta":586235,"kb_avail":93748584,"statfs":{"total":96636764160,"available":95998550016,"internally_reserved":0,"allocated":37896192,"data_stored":37712371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":600305217},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":12.882999999999999},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":12.271000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.21},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":9.2989999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.2409999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":959176,"kb_used_data":35752,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93412664,"statfs":{"total":96636764160,"available":95654567936,"internally_reserved":0,"allocated":36610048,"data_stored":36426227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.307},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":4.952}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.58099999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:15:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.617,"15min":1.319},"min":{"1min":0.26500000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":11.708,"5min":20.199000000000002,"15min":20.948},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.3340000000000001,"5min":2.3180000000000001,"15min":1.4259999999999999},"min":{"1min":0.27900000000000003,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":11.734,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.38900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36265984,"data_stored":36265984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":37478400,"data_stored":37478400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36392960,"data_stored":36392960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36372480,"data_stored":36372480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:56.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:16:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:16:58.677 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:16:58.704 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-15T03:16:56.966399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":65870420,"num_objects":8046,"num_object_clones":0,"num_object_copies":32184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182661,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7436,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160888,"ondisk_log_size":160888,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3491052,"kb_used_data":130888,"kb_used_omap":14,"kb_used_meta":3360113,"kb_avail":373996308,"statfs":{"total":386547056640,"available":382972219392,"internally_reserved":0,"allocated":134029312,"data_stored":133297174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3440755739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,2,2,11,16,12,9,7,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21479424,"num_objects":-2622,"num_object_clones":0,"num_object_copies":-10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2622,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002921"},"pg_stats":[{"pgid":"1.f","version":"37'11400","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.505819+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:56.505819+0000","last_peered":"2024-09-15T03:16:56.505819+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:56.505819+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4177920,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11400,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10938","reported_seq":13435,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.717479+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:56.717479+0000","last_peered":"2024-09-15T03:16:56.717479+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:56.717479+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3995250,"num_objects":488,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11700","reported_seq":11809,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.546058+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:56.546058+0000","last_peered":"2024-09-15T03:16:56.546058+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:56.546058+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4258641,"num_objects":521,"num_object_clones":0,"num_object_copies":2084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11699,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1570}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11137","reported_seq":15109,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:55.199253+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:16:55.199253+0000","last_peered":"2024-09-15T03:16:55.199253+0000","last_clean":"2024-09-15T03:16:55.199253+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:16:55.199253+0000","last_undegraded":"2024-09-15T03:16:55.199253+0000","last_fullsized":"2024-09-15T03:16:55.199253+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4019826,"num_objects":491,"num_object_clones":0,"num_object_copies":1964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11135,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11572","reported_seq":11679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.281436+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:56.281436+0000","last_peered":"2024-09-15T03:16:56.281436+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:56.281436+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4167282,"num_objects":509,"num_object_clones":0,"num_object_copies":2036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11570,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11515","reported_seq":11600,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.595511+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:56.595511+0000","last_peered":"2024-09-15T03:16:56.595511+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:56.595511+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3956736,"num_objects":483,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11515,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11462","reported_seq":11584,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.748472+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:56.748472+0000","last_peered":"2024-09-15T03:16:56.748472+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:56.748472+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4060786,"num_objects":496,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1530}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11607","reported_seq":11726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.584217+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:56.584217+0000","last_peered":"2024-09-15T03:16:56.584217+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:56.584217+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3913330,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11463","reported_seq":11555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.708333+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:56.708333+0000","last_peered":"2024-09-15T03:16:56.708333+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:56.708333+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4148452,"num_objects":507,"num_object_clones":0,"num_object_copies":2028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11449","reported_seq":11570,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.362180+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:56.362180+0000","last_peered":"2024-09-15T03:16:56.362180+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:56.362180+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4249202,"num_objects":519,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1523}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11366","reported_seq":11462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.650069+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:56.650069+0000","last_peered":"2024-09-15T03:16:56.650069+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:56.650069+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4231595,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11363,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11388","reported_seq":15431,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:55.108611+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:16:55.108611+0000","last_peered":"2024-09-15T03:16:55.108611+0000","last_clean":"2024-09-15T03:16:55.108611+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:16:55.108611+0000","last_undegraded":"2024-09-15T03:16:55.108611+0000","last_fullsized":"2024-09-15T03:16:55.108611+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4184889,"num_objects":511,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11387,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11602","reported_seq":15778,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:16:55.196645+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:16:55.196645+0000","last_peered":"2024-09-15T03:16:55.196645+0000","last_clean":"2024-09-15T03:16:55.196645+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:16:55.196645+0000","last_undegraded":"2024-09-15T03:16:55.196645+0000","last_fullsized":"2024-09-15T03:16:55.196645+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348729,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11601,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11189","reported_seq":11291,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.649968+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:56.649968+0000","last_peered":"2024-09-15T03:16:56.649968+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:56.649968+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4036210,"num_objects":493,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1520}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11380","reported_seq":11489,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.509499+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:56.509499+0000","last_peered":"2024-09-15T03:16:56.509499+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:56.509499+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4118130,"num_objects":503,"num_object_clones":0,"num_object_copies":2012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11520","reported_seq":11635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.486363+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:56.486363+0000","last_peered":"2024-09-15T03:16:56.486363+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:56.486363+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4003442,"num_objects":489,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":65870420,"num_objects":8046,"num_object_clones":0,"num_object_copies":32184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182661,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7436,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131936256,"data_stored":131936256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160888,"ondisk_log_size":160888,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951992,"kb_used_data":32752,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93419848,"statfs":{"total":96636764160,"available":95661924352,"internally_reserved":0,"allocated":33538048,"data_stored":33354227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,8,1,4,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":11.140000000000001},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":11.006}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.252},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.80900000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":955424,"kb_used_data":32468,"kb_used_omap":1,"kb_used_meta":922942,"kb_avail":93416416,"statfs":{"total":96636764160,"available":95658409984,"internally_reserved":0,"allocated":33247232,"data_stored":33066557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945092952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":4.1390000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.28999999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.8889999999999998},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":3.927}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":3.9670000000000001},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":3.976}]}]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623884,"kb_used_data":33476,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93747956,"statfs":{"total":96636764160,"available":95997906944,"internally_reserved":0,"allocated":34279424,"data_stored":34095603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":12.882999999999999},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":12.271000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.21},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":9.2989999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.2409999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":959752,"kb_used_data":32192,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93412088,"statfs":{"total":96636764160,"available":95653978112,"internally_reserved":0,"allocated":32964608,"data_stored":32780787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,2,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.27400000000000002},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.435}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":10.321999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.378}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32608256,"data_stored":32608256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":33902592,"data_stored":33902592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":32739328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32686080,"data_stored":32686080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:16:58.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:16:58.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11400", 'reported_seq': 11468, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.505819+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:16:56.505819+0000', 'last_peered': '2024-09-15T03:16:56.505819+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:16:56.505819+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4177920, 'num_objects': 510, 'num_object_clones': 0, 'num_object_copies': 2040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11400, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1456}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:58.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:16:58.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'10938", 'reported_seq': 13435, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.717479+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:16:56.717479+0000', 'last_peered': '2024-09-15T03:16:56.717479+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:16:56.717479+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3995250, 'num_objects': 488, 'num_object_clones': 0, 'num_object_copies': 1952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 786, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10936, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1216, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -298}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:58.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:16:58.706 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11700", 'reported_seq': 11809, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.546058+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:16:56.546058+0000', 'last_peered': '2024-09-15T03:16:56.546058+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:16:56.546058+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4258641, 'num_objects': 521, 'num_object_clones': 0, 'num_object_copies': 2084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 521, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11699, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1570}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:58.706 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11572", 'reported_seq': 11679, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.281436+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:16:56.281436+0000', 'last_peered': '2024-09-15T03:16:56.281436+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:16:56.281436+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4167282, 'num_objects': 509, 'num_object_clones': 0, 'num_object_copies': 2036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11570, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1575}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11515", 'reported_seq': 11600, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.595511+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:16:56.595511+0000', 'last_peered': '2024-09-15T03:16:56.595511+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:16:56.595511+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3956736, 'num_objects': 483, 'num_object_clones': 0, 'num_object_copies': 1932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11515, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1555}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11462", 'reported_seq': 11584, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.748472+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:16:56.748472+0000', 'last_peered': '2024-09-15T03:16:56.748472+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:16:56.748472+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4060786, 'num_objects': 496, 'num_object_clones': 0, 'num_object_copies': 1984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 496, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11460, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1530}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11607", 'reported_seq': 11726, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.584217+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:16:56.584217+0000', 'last_peered': '2024-09-15T03:16:56.584217+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:16:56.584217+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3913330, 'num_objects': 478, 'num_object_clones': 0, 'num_object_copies': 1912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11605, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:16:58.707 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11463", 'reported_seq': 11555, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.708333+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:16:56.708333+0000', 'last_peered': '2024-09-15T03:16:56.708333+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:16:56.708333+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4148452, 'num_objects': 507, 'num_object_clones': 0, 'num_object_copies': 2028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11460, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1520}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11449", 'reported_seq': 11570, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.362180+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:16:56.362180+0000', 'last_peered': '2024-09-15T03:16:56.362180+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:16:56.362180+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4249202, 'num_objects': 519, 'num_object_clones': 0, 'num_object_copies': 2076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11447, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1523}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11366", 'reported_seq': 11462, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.650069+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:16:56.650069+0000', 'last_peered': '2024-09-15T03:16:56.650069+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:16:56.650069+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4231595, 'num_objects': 517, 'num_object_clones': 0, 'num_object_copies': 2068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11363, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1545}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11189", 'reported_seq': 11291, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.649968+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:16:56.649968+0000', 'last_peered': '2024-09-15T03:16:56.649968+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:16:56.649968+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4036210, 'num_objects': 493, 'num_object_clones': 0, 'num_object_copies': 1972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11187, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1520}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11380", 'reported_seq': 11489, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.509499+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:16:56.509499+0000', 'last_peered': '2024-09-15T03:16:56.509499+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:16:56.509499+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4118130, 'num_objects': 503, 'num_object_clones': 0, 'num_object_copies': 2012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11378, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1474}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:16:58.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:16:58.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11520", 'reported_seq': 11635, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:16:56.486363+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:16:56.486363+0000', 'last_peered': '2024-09-15T03:16:56.486363+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:16:56.486363+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4003442, 'num_objects': 489, 'num_object_clones': 0, 'num_object_copies': 1956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11518, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1570}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:16:58.709 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:17:00.701 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:00.701 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25248/32184 objects degraded (78.449%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":743,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8046,"data_bytes":65870420,"bytes_used":3574837248,"bytes_avail":382972219392,"bytes_total":386547056640,"degraded_objects":25248,"degraded_total":32184,"degraded_ratio":0.78448918717375093,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":144},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:17:00.756 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:17:00.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:02.442 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:02.473 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-15T03:17:00.967314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":64354900,"num_objects":7861,"num_object_clones":0,"num_object_copies":31444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7861,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182846,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7436,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3491852,"kb_used_data":127524,"kb_used_omap":14,"kb_used_meta":3364273,"kb_avail":373995508,"statfs":{"total":386547056640,"available":382971400192,"internally_reserved":0,"allocated":130584576,"data_stored":129852438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3445015579},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,2,2,11,18,14,10,8,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15769600,"num_objects":-1925,"num_object_clones":0,"num_object_copies":-7700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1925,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002848"},"pg_stats":[{"pgid":"1.f","version":"37'11400","reported_seq":11468,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.505819+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:16:56.505819+0000","last_peered":"2024-09-15T03:16:56.505819+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:16:56.505819+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4177920,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11400,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1456}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'10938","reported_seq":13435,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.717479+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:16:56.717479+0000","last_peered":"2024-09-15T03:16:56.717479+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:16:56.717479+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3995250,"num_objects":488,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-298}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11700","reported_seq":11809,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.546058+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:16:56.546058+0000","last_peered":"2024-09-15T03:16:56.546058+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:16:56.546058+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4258641,"num_objects":521,"num_object_clones":0,"num_object_copies":2084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11699,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1570}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11189","reported_seq":15163,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:00.307099+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:00.307099+0000","last_peered":"2024-09-15T03:17:00.307099+0000","last_clean":"2024-09-15T03:17:00.307099+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:00.307099+0000","last_undegraded":"2024-09-15T03:17:00.307099+0000","last_fullsized":"2024-09-15T03:17:00.307099+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3593842,"num_objects":439,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11572","reported_seq":11679,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.281436+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:16:56.281436+0000","last_peered":"2024-09-15T03:16:56.281436+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:16:56.281436+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4167282,"num_objects":509,"num_object_clones":0,"num_object_copies":2036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11570,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1575}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11515","reported_seq":11600,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.595511+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:16:56.595511+0000","last_peered":"2024-09-15T03:16:56.595511+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:16:56.595511+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3956736,"num_objects":483,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11515,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11462","reported_seq":11584,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.748472+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:16:56.748472+0000","last_peered":"2024-09-15T03:16:56.748472+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:16:56.748472+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4060786,"num_objects":496,"num_object_clones":0,"num_object_copies":1984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1530}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11607","reported_seq":11726,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.584217+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:16:56.584217+0000","last_peered":"2024-09-15T03:16:56.584217+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:16:56.584217+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3913330,"num_objects":478,"num_object_clones":0,"num_object_copies":1912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1578}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11463","reported_seq":11555,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.708333+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:16:56.708333+0000","last_peered":"2024-09-15T03:16:56.708333+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:16:56.708333+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4148452,"num_objects":507,"num_object_clones":0,"num_object_copies":2028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1520}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11449","reported_seq":11570,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.362180+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:16:56.362180+0000","last_peered":"2024-09-15T03:16:56.362180+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:16:56.362180+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4249202,"num_objects":519,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11447,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1523}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11366","reported_seq":11462,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.650069+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:16:56.650069+0000","last_peered":"2024-09-15T03:16:56.650069+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:16:56.650069+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4231595,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11363,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1545}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11460","reported_seq":15503,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:00.267482+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:00.267482+0000","last_peered":"2024-09-15T03:17:00.267482+0000","last_clean":"2024-09-15T03:17:00.267482+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:00.267482+0000","last_undegraded":"2024-09-15T03:17:00.267482+0000","last_fullsized":"2024-09-15T03:17:00.267482+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3595065,"num_objects":439,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11663","reported_seq":15839,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:00.391860+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:00.391860+0000","last_peered":"2024-09-15T03:17:00.391860+0000","last_clean":"2024-09-15T03:17:00.391860+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:00.391860+0000","last_undegraded":"2024-09-15T03:17:00.391860+0000","last_fullsized":"2024-09-15T03:17:00.391860+0000","mapping_epoch":36,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3849017,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11662,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11189","reported_seq":11291,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.649968+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:16:56.649968+0000","last_peered":"2024-09-15T03:16:56.649968+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:16:56.649968+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4036210,"num_objects":493,"num_object_clones":0,"num_object_copies":1972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11187,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1520}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11380","reported_seq":11489,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.509499+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:16:56.509499+0000","last_peered":"2024-09-15T03:16:56.509499+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:16:56.509499+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4118130,"num_objects":503,"num_object_clones":0,"num_object_copies":2012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11378,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1474}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11520","reported_seq":11635,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:16:56.486363+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:16:56.486363+0000","last_peered":"2024-09-15T03:16:56.486363+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:16:56.486363+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4003442,"num_objects":489,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11518,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":64354900,"num_objects":7861,"num_object_clones":0,"num_object_copies":31444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7861,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182846,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7436,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":128335872,"data_stored":128335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160973,"ondisk_log_size":160973,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":951992,"kb_used_data":32752,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93419848,"statfs":{"total":96636764160,"available":95661924352,"internally_reserved":0,"allocated":33538048,"data_stored":33354227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,3,8,1,4,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":11.140000000000001},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":11.006}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.252},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.28599999999999998},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.80900000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":955424,"kb_used_data":32468,"kb_used_omap":1,"kb_used_meta":922942,"kb_avail":93416416,"statfs":{"total":96636764160,"available":95658409984,"internally_reserved":0,"allocated":33247232,"data_stored":33066557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945092952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":4.1390000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.28999999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":3.8889999999999998},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":3.927}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":3.9670000000000001},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":3.976}]}]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624684,"kb_used_data":30112,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93747156,"statfs":{"total":96636764160,"available":95997087744,"internally_reserved":0,"allocated":30834688,"data_stored":30650867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,5,9,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":6.4249999999999998},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.249}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.29499999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":959752,"kb_used_data":32192,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93412088,"statfs":{"total":96636764160,"available":95653978112,"internally_reserved":0,"allocated":32964608,"data_stored":32780787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,2,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.27400000000000002},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.435}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":10.321999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.378}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32608256,"data_stored":32608256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30302208,"data_stored":30302208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32739328,"data_stored":32739328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32686080,"data_stored":32686080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:05.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:07.532 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:07.546 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-15T03:17:06.968758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51444308,"num_objects":6285,"num_object_clones":0,"num_object_copies":25140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184422,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7544,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160649,"ondisk_log_size":160649,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3471808,"kb_used_data":103328,"kb_used_omap":14,"kb_used_meta":3368433,"kb_avail":374015552,"statfs":{"total":386547056640,"available":382991925248,"internally_reserved":0,"allocated":105807872,"data_stored":105075734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3449275419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,13,1,12,15,12,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21667840,"num_objects":-2645,"num_object_clones":0,"num_object_copies":-10580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2645,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002825"},"pg_stats":[{"pgid":"1.f","version":"37'11507","reported_seq":11575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.602426+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:06.602426+0000","last_peered":"2024-09-15T03:17:06.602426+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:06.602426+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3301376,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11043","reported_seq":13756,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.770686+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:06.770686+0000","last_peered":"2024-09-15T03:17:06.770686+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:06.770686+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3135090,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11819","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.430292+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:06.430292+0000","last_peered":"2024-09-15T03:17:06.430292+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:06.430292+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3283793,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11818,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11246","reported_seq":15220,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:05.516954+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:05.516954+0000","last_peered":"2024-09-15T03:17:05.516954+0000","last_clean":"2024-09-15T03:17:05.516954+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:05.516954+0000","last_undegraded":"2024-09-15T03:17:05.516954+0000","last_fullsized":"2024-09-15T03:17:05.516954+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3126898,"num_objects":382,"num_object_clones":0,"num_object_copies":1528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11698","reported_seq":11806,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.852498+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:06.852498+0000","last_peered":"2024-09-15T03:17:06.852498+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:06.852498+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3135090,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11696,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11609","reported_seq":11695,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.507158+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:06.507158+0000","last_peered":"2024-09-15T03:17:06.507158+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:06.507158+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3186688,"num_objects":389,"num_object_clones":0,"num_object_copies":1556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11609,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11567","reported_seq":11690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.572002+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:06.572002+0000","last_peered":"2024-09-15T03:17:06.572002+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:06.572002+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3200626,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11565,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1635}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11702","reported_seq":11821,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.525026+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:06.525026+0000","last_peered":"2024-09-15T03:17:06.525026+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:06.525026+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3135090,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11700,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11567","reported_seq":11660,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.694492+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:06.694492+0000","last_peered":"2024-09-15T03:17:06.694492+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:06.694492+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3296484,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11564,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1624}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11579","reported_seq":11699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.605775+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:06.605775+0000","last_peered":"2024-09-15T03:17:06.605775+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:06.605775+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3184242,"num_objects":389,"num_object_clones":0,"num_object_copies":1556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11577,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1653}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11485","reported_seq":11581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.769983+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:06.769983+0000","last_peered":"2024-09-15T03:17:06.769983+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:06.769983+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3256747,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11508","reported_seq":15551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:05.202215+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:05.202215+0000","last_peered":"2024-09-15T03:17:05.202215+0000","last_clean":"2024-09-15T03:17:05.202215+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:05.202215+0000","last_undegraded":"2024-09-15T03:17:05.202215+0000","last_fullsized":"2024-09-15T03:17:05.202215+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3201849,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11712","reported_seq":15888,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:05.577711+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:05.577711+0000","last_peered":"2024-09-15T03:17:05.577711+0000","last_clean":"2024-09-15T03:17:05.577711+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:05.577711+0000","last_undegraded":"2024-09-15T03:17:05.577711+0000","last_fullsized":"2024-09-15T03:17:05.577711+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3447609,"num_objects":421,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11297","reported_seq":11399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.822613+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:06.822613+0000","last_peered":"2024-09-15T03:17:06.822613+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:06.822613+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3151474,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11502","reported_seq":11607,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.671066+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:06.671066+0000","last_peered":"2024-09-15T03:17:06.671066+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:06.671066+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3118706,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11500,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11608","reported_seq":11723,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.732318+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:06.732318+0000","last_peered":"2024-09-15T03:17:06.732318+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:06.732318+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3282546,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11606,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":51444308,"num_objects":6285,"num_object_clones":0,"num_object_copies":25140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184422,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7544,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":103063552,"data_stored":103063552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160649,"ondisk_log_size":160649,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":956952,"kb_used_data":25232,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93414888,"statfs":{"total":96636764160,"available":95656845312,"internally_reserved":0,"allocated":25837568,"data_stored":25653747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,5,4,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":4.1959999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":4.7210000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":3.556}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":3.976},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.39600000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944376,"kb_used_data":25584,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93427464,"statfs":{"total":96636764160,"available":95669723136,"internally_reserved":0,"allocated":26198016,"data_stored":26017341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,2,8,2,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.45400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":9.2240000000000002},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":5.2770000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625332,"kb_used_data":26604,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93746508,"statfs":{"total":96636764160,"available":95996424192,"internally_reserved":0,"allocated":27242496,"data_stored":27058675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":28.456},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":13.598000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.32400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":945148,"kb_used_data":25908,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93426692,"statfs":{"total":96636764160,"available":95668932608,"internally_reserved":0,"allocated":26529792,"data_stored":26345971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.436},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":5.1109999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":5.0789999999999997},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.28000000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":4.3899999999999997},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.316}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25473024,"data_stored":25473024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26603520,"data_stored":26603520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25530368,"data_stored":25530368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25456640,"data_stored":25456640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:07.547 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:09.215 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:09.220 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:09.281 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-15T03:17:08.969272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51444308,"num_objects":6285,"num_object_clones":0,"num_object_copies":25140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184422,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7544,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160649,"ondisk_log_size":160649,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3471808,"kb_used_data":103328,"kb_used_omap":14,"kb_used_meta":3368433,"kb_avail":374015552,"statfs":{"total":386547056640,"available":382991925248,"internally_reserved":0,"allocated":105807872,"data_stored":105075734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3449275419},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,13,1,12,15,12,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14426112,"num_objects":-1761,"num_object_clones":0,"num_object_copies":-7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002872"},"pg_stats":[{"pgid":"1.f","version":"37'11507","reported_seq":11575,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.602426+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:06.602426+0000","last_peered":"2024-09-15T03:17:06.602426+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:06.602426+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3301376,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1563}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11043","reported_seq":13756,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.770686+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:06.770686+0000","last_peered":"2024-09-15T03:17:06.770686+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:06.770686+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3135090,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":678,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11041,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-295}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11819","reported_seq":11929,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.430292+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:06.430292+0000","last_peered":"2024-09-15T03:17:06.430292+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:06.430292+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3283793,"num_objects":402,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11818,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11246","reported_seq":15220,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:05.516954+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:05.516954+0000","last_peered":"2024-09-15T03:17:05.516954+0000","last_clean":"2024-09-15T03:17:05.516954+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:05.516954+0000","last_undegraded":"2024-09-15T03:17:05.516954+0000","last_fullsized":"2024-09-15T03:17:05.516954+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3126898,"num_objects":382,"num_object_clones":0,"num_object_copies":1528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11698","reported_seq":11806,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.852498+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:06.852498+0000","last_peered":"2024-09-15T03:17:06.852498+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:06.852498+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3135090,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11696,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1701}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11609","reported_seq":11695,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.507158+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:06.507158+0000","last_peered":"2024-09-15T03:17:06.507158+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:06.507158+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3186688,"num_objects":389,"num_object_clones":0,"num_object_copies":1556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11609,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11567","reported_seq":11690,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.572002+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:06.572002+0000","last_peered":"2024-09-15T03:17:06.572002+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:06.572002+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3200626,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11565,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1635}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11702","reported_seq":11821,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.525026+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:06.525026+0000","last_peered":"2024-09-15T03:17:06.525026+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:06.525026+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3135090,"num_objects":383,"num_object_clones":0,"num_object_copies":1532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11700,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11567","reported_seq":11660,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.694492+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:06.694492+0000","last_peered":"2024-09-15T03:17:06.694492+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:06.694492+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3296484,"num_objects":403,"num_object_clones":0,"num_object_copies":1612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11564,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1624}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11579","reported_seq":11699,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.605775+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:06.605775+0000","last_peered":"2024-09-15T03:17:06.605775+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:06.605775+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3184242,"num_objects":389,"num_object_clones":0,"num_object_copies":1556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11577,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1653}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11485","reported_seq":11581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.769983+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:06.769983+0000","last_peered":"2024-09-15T03:17:06.769983+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:06.769983+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3256747,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11508","reported_seq":15551,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:05.202215+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:05.202215+0000","last_peered":"2024-09-15T03:17:05.202215+0000","last_clean":"2024-09-15T03:17:05.202215+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:05.202215+0000","last_undegraded":"2024-09-15T03:17:05.202215+0000","last_fullsized":"2024-09-15T03:17:05.202215+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3201849,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11712","reported_seq":15888,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:05.577711+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:05.577711+0000","last_peered":"2024-09-15T03:17:05.577711+0000","last_clean":"2024-09-15T03:17:05.577711+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:05.577711+0000","last_undegraded":"2024-09-15T03:17:05.577711+0000","last_fullsized":"2024-09-15T03:17:05.577711+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3447609,"num_objects":421,"num_object_clones":0,"num_object_copies":1684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11711,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11297","reported_seq":11399,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.822613+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:06.822613+0000","last_peered":"2024-09-15T03:17:06.822613+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:06.822613+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3151474,"num_objects":385,"num_object_clones":0,"num_object_copies":1540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11502","reported_seq":11607,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.671066+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:06.671066+0000","last_peered":"2024-09-15T03:17:06.671066+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:06.671066+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3118706,"num_objects":381,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11500,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1596}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11608","reported_seq":11723,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:06.732318+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:06.732318+0000","last_peered":"2024-09-15T03:17:06.732318+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:06.732318+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3282546,"num_objects":401,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11606,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":51444308,"num_objects":6285,"num_object_clones":0,"num_object_copies":25140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184422,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7544,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":103063552,"data_stored":103063552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160649,"ondisk_log_size":160649,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":956952,"kb_used_data":25232,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93414888,"statfs":{"total":96636764160,"available":95656845312,"internally_reserved":0,"allocated":25837568,"data_stored":25653747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,5,4,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":4.1959999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":4.7210000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":3.556}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":3.976},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.39600000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944376,"kb_used_data":25584,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93427464,"statfs":{"total":96636764160,"available":95669723136,"internally_reserved":0,"allocated":26198016,"data_stored":26017341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,0,2,8,2,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.45400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":9.2240000000000002},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":5.2770000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625332,"kb_used_data":26604,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93746508,"statfs":{"total":96636764160,"available":95996424192,"internally_reserved":0,"allocated":27242496,"data_stored":27058675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,3,1,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":28.456},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":13.598000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.32400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":945148,"kb_used_data":25908,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93426692,"statfs":{"total":96636764160,"available":95668932608,"internally_reserved":0,"allocated":26529792,"data_stored":26345971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.436},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":5.1109999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":5.0789999999999997},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.28000000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":4.3899999999999997},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.316}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25473024,"data_stored":25473024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26603520,"data_stored":26603520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25530368,"data_stored":25530368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":25456640,"data_stored":25456640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:09.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:17:09.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11507", 'reported_seq': 11575, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.602426+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:06.602426+0000', 'last_peered': '2024-09-15T03:17:06.602426+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:06.602426+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3301376, 'num_objects': 403, 'num_object_clones': 0, 'num_object_copies': 1612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11507, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1563}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:09.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:17:09.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11043", 'reported_seq': 13756, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.770686+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:17:06.770686+0000', 'last_peered': '2024-09-15T03:17:06.770686+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:17:06.770686+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3135090, 'num_objects': 383, 'num_object_clones': 0, 'num_object_copies': 1532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 678, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11041, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1324, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -295}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11819", 'reported_seq': 11929, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.430292+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:06.430292+0000', 'last_peered': '2024-09-15T03:17:06.430292+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:06.430292+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3283793, 'num_objects': 402, 'num_object_clones': 0, 'num_object_copies': 1608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 402, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11818, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1689}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11698", 'reported_seq': 11806, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.852498+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:06.852498+0000', 'last_peered': '2024-09-15T03:17:06.852498+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:06.852498+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3135090, 'num_objects': 383, 'num_object_clones': 0, 'num_object_copies': 1532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11696, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1701}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11609", 'reported_seq': 11695, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.507158+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:06.507158+0000', 'last_peered': '2024-09-15T03:17:06.507158+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:06.507158+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3186688, 'num_objects': 389, 'num_object_clones': 0, 'num_object_copies': 1556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11609, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1649}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11567", 'reported_seq': 11690, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.572002+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:06.572002+0000', 'last_peered': '2024-09-15T03:17:06.572002+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:06.572002+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3200626, 'num_objects': 391, 'num_object_clones': 0, 'num_object_copies': 1564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11565, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1635}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11702", 'reported_seq': 11821, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.525026+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:06.525026+0000', 'last_peered': '2024-09-15T03:17:06.525026+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:06.525026+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3135090, 'num_objects': 383, 'num_object_clones': 0, 'num_object_copies': 1532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11700, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1673}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:09.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11567", 'reported_seq': 11660, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.694492+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:17:06.694492+0000', 'last_peered': '2024-09-15T03:17:06.694492+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:17:06.694492+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3296484, 'num_objects': 403, 'num_object_clones': 0, 'num_object_copies': 1612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11564, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1624}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11579", 'reported_seq': 11699, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.605775+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:06.605775+0000', 'last_peered': '2024-09-15T03:17:06.605775+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:06.605775+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3184242, 'num_objects': 389, 'num_object_clones': 0, 'num_object_copies': 1556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11577, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1653}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11485", 'reported_seq': 11581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.769983+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:06.769983+0000', 'last_peered': '2024-09-15T03:17:06.769983+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:06.769983+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3256747, 'num_objects': 398, 'num_object_clones': 0, 'num_object_copies': 1592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11482, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1664}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11297", 'reported_seq': 11399, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.822613+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:06.822613+0000', 'last_peered': '2024-09-15T03:17:06.822613+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:06.822613+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3151474, 'num_objects': 385, 'num_object_clones': 0, 'num_object_copies': 1540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11295, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1628}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11502", 'reported_seq': 11607, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.671066+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:06.671066+0000', 'last_peered': '2024-09-15T03:17:06.671066+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:06.671066+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3118706, 'num_objects': 381, 'num_object_clones': 0, 'num_object_copies': 1524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11500, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1596}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:17:09.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11608", 'reported_seq': 11723, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:06.732318+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:06.732318+0000', 'last_peered': '2024-09-15T03:17:06.732318+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:06.732318+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3282546, 'num_objects': 401, 'num_object_clones': 0, 'num_object_copies': 1604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11606, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1658}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:09.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:17:11.163 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:11.164 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25140/25140 objects degraded (100.000%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":754,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6285,"data_bytes":51444308,"bytes_used":3555131392,"bytes_avail":382991925248,"bytes_total":386547056640,"degraded_objects":25140,"degraded_total":25140,"degraded_ratio":1,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":146},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:17:11.212 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:17:11.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:13.207 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:13.231 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:13.299 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-15T03:17:12.972406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":44268116,"num_objects":5409,"num_object_clones":0,"num_object_copies":21636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185298,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7599,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3449332,"kb_used_data":89172,"kb_used_omap":14,"kb_used_meta":3360113,"kb_avail":374038028,"statfs":{"total":386547056640,"available":383014940672,"internally_reserved":0,"allocated":91312128,"data_stored":90579990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3440755739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,6,12,9,6,12,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20086784,"num_objects":-2452,"num_object_clones":0,"num_object_copies":-9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005091"},"pg_stats":[{"pgid":"1.f","version":"37'11569","reported_seq":11637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.577903+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:11.577903+0000","last_peered":"2024-09-15T03:17:11.577903+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:11.577903+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2793472,"num_objects":341,"num_object_clones":0,"num_object_copies":1364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11569,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1625}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11095","reported_seq":13918,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.784524+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:11.784524+0000","last_peered":"2024-09-15T03:17:11.784524+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:11.784524+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2709106,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":623,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11093,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-292}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11866","reported_seq":11976,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.610276+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:11.610276+0000","last_peered":"2024-09-15T03:17:11.610276+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:11.610276+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2898769,"num_objects":355,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":355,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11865,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1736}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11296","reported_seq":15270,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:10.508502+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:10.508502+0000","last_peered":"2024-09-15T03:17:10.508502+0000","last_clean":"2024-09-15T03:17:10.508502+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:10.508502+0000","last_undegraded":"2024-09-15T03:17:10.508502+0000","last_fullsized":"2024-09-15T03:17:10.508502+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2717298,"num_objects":332,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11294,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11751","reported_seq":11859,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.757810+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:11.757810+0000","last_peered":"2024-09-15T03:17:11.757810+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:11.757810+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2700914,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11669","reported_seq":11755,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.675980+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:11.675980+0000","last_peered":"2024-09-15T03:17:11.675980+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:11.675980+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2695168,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11613","reported_seq":11736,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.488665+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:11.488665+0000","last_peered":"2024-09-15T03:17:11.488665+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:11.488665+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823794,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11611,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1681}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11766","reported_seq":11885,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.808670+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:11.808670+0000","last_peered":"2024-09-15T03:17:11.808670+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:11.808670+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2610802,"num_objects":319,"num_object_clones":0,"num_object_copies":1276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11764,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11628","reported_seq":11721,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.820574+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:11.820574+0000","last_peered":"2024-09-15T03:17:11.820574+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:11.820574+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2796772,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11625,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11623","reported_seq":11743,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.713792+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:11.713792+0000","last_peered":"2024-09-15T03:17:11.713792+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:11.713792+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2823794,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11621,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1697}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11538","reported_seq":11634,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.560689+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:11.560689+0000","last_peered":"2024-09-15T03:17:11.560689+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:11.560689+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822571,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11535,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11557","reported_seq":15600,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:10.568896+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:10.568896+0000","last_peered":"2024-09-15T03:17:10.568896+0000","last_clean":"2024-09-15T03:17:10.568896+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:10.568896+0000","last_undegraded":"2024-09-15T03:17:10.568896+0000","last_fullsized":"2024-09-15T03:17:10.568896+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2800441,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11776","reported_seq":15952,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:10.525076+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:10.525076+0000","last_peered":"2024-09-15T03:17:10.525076+0000","last_clean":"2024-09-15T03:17:10.525076+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:10.525076+0000","last_undegraded":"2024-09-15T03:17:10.525076+0000","last_fullsized":"2024-09-15T03:17:10.525076+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2923321,"num_objects":357,"num_object_clones":0,"num_object_copies":1428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11358","reported_seq":11460,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.868380+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:11.868380+0000","last_peered":"2024-09-15T03:17:11.868380+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:11.868380+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2651762,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11356,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1689}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11552","reported_seq":11661,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.480650+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:11.480650+0000","last_peered":"2024-09-15T03:17:11.480650+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:11.480650+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2709106,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11550,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1646}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11668","reported_seq":11783,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.864517+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:11.864517+0000","last_peered":"2024-09-15T03:17:11.864517+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:11.864517+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2791026,"num_objects":341,"num_object_clones":0,"num_object_copies":1364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1718}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":44268116,"num_objects":5409,"num_object_clones":0,"num_object_copies":21636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5409,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185298,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7599,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":88719360,"data_stored":88719360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":940992,"kb_used_data":21752,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93430848,"statfs":{"total":96636764160,"available":95673188352,"internally_reserved":0,"allocated":22274048,"data_stored":22090227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,5,5,4,1,3,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.34100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.53200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":945004,"kb_used_data":22052,"kb_used_omap":1,"kb_used_meta":922942,"kb_avail":93426836,"statfs":{"total":96636764160,"available":95669080064,"internally_reserved":0,"allocated":22581248,"data_stored":22400573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945092952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,0,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.251},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":4.8680000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":1.046}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":4.915}]}]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613436,"kb_used_data":23028,"kb_used_omap":4,"kb_used_meta":590395,"kb_avail":93758404,"statfs":{"total":96636764160,"available":96008605696,"internally_reserved":0,"allocated":23580672,"data_stored":23396851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":604565057},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,3,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.903,"5min":3.633,"15min":2.3519999999999999},"min":{"1min":0.183,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":15.23,"5min":28.076000000000001,"15min":28.076000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.859,"5min":2.573,"15min":1.954},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.187},"max":{"1min":13.845000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":10.4}]},{"osd":2,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":2.278,"5min":3.081,"15min":2.1190000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":11.462,"5min":25.661000000000001,"15min":26.911000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":3.145,"5min":4.492,"15min":2.7330000000000001},"min":{"1min":0.27000000000000002,"5min":0.20200000000000001,"15min":0.20000000000000001},"max":{"1min":8.5470000000000006,"5min":28.145,"15min":28.145},"last":0.26400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:15 2024","interfaces":[{"interface":"back","average":{"1min":6.1479999999999997,"5min":4.2000000000000002,"15min":2.5099999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.222},"max":{"1min":24.797000000000001,"5min":59.225999999999999,"15min":59.225999999999999},"last":12.708},{"interface":"front","average":{"1min":2.0059999999999998,"5min":2.3380000000000001,"15min":1.71},"min":{"1min":0.25800000000000001,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":6.8280000000000003,"5min":33.444000000000003,"15min":33.444000000000003},"last":10.821}]}]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":949900,"kb_used_data":22340,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93421940,"statfs":{"total":96636764160,"available":95664066560,"internally_reserved":0,"allocated":22876160,"data_stored":22692339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":4.5819999999999999},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":4.5199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":4.6879999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":4.6500000000000004},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":12.644}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21872640,"data_stored":21872640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":22962176,"data_stored":22962176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":22011904,"data_stored":22011904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21872640,"data_stored":21872640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:16.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:17.949 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:17.997 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:18.007 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-15T03:17:16.975132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39598676,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185868,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7638,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3433828,"kb_used_data":77824,"kb_used_omap":14,"kb_used_meta":3355953,"kb_avail":374053532,"statfs":{"total":386547056640,"available":383030816768,"internally_reserved":0,"allocated":79691776,"data_stored":78959638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3436495899},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,7,11,9,12,9,14],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19079168,"num_objects":-2329,"num_object_clones":0,"num_object_copies":-9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2329,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006847"},"pg_stats":[{"pgid":"1.f","version":"37'11569","reported_seq":11637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.577903+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:11.577903+0000","last_peered":"2024-09-15T03:17:11.577903+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:11.577903+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2793472,"num_objects":341,"num_object_clones":0,"num_object_copies":1364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11569,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1625}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11164","reported_seq":14065,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.861985+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:16.861985+0000","last_peered":"2024-09-15T03:17:16.861985+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:16.861985+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2143858,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11162,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11928","reported_seq":12038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.740499+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:16.740499+0000","last_peered":"2024-09-15T03:17:16.740499+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:16.740499+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2390865,"num_objects":293,"num_object_clones":0,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11927,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1798}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11340","reported_seq":15313,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:15.821734+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:15.821734+0000","last_peered":"2024-09-15T03:17:15.821734+0000","last_clean":"2024-09-15T03:17:15.821734+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:15.821734+0000","last_undegraded":"2024-09-15T03:17:15.821734+0000","last_fullsized":"2024-09-15T03:17:15.821734+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2356850,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11338,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11751","reported_seq":11859,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.757810+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:11.757810+0000","last_peered":"2024-09-15T03:17:11.757810+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:11.757810+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2700914,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11669","reported_seq":11755,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.675980+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:11.675980+0000","last_peered":"2024-09-15T03:17:11.675980+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:11.675980+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2695168,"num_objects":329,"num_object_clones":0,"num_object_copies":1316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11669,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1709}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11667","reported_seq":11790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.708220+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:16.708220+0000","last_peered":"2024-09-15T03:17:16.708220+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:16.708220+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2381426,"num_objects":291,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11665,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1735}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11812","reported_seq":11931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.851635+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:16.851635+0000","last_peered":"2024-09-15T03:17:16.851635+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:16.851635+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2233970,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11628","reported_seq":11721,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.820574+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:11.820574+0000","last_peered":"2024-09-15T03:17:11.820574+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:11.820574+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2796772,"num_objects":342,"num_object_clones":0,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11625,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11684","reported_seq":11805,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.691782+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:16.691782+0000","last_peered":"2024-09-15T03:17:16.691782+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:16.691782+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2324082,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11538","reported_seq":11634,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.560689+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:11.560689+0000","last_peered":"2024-09-15T03:17:11.560689+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:11.560689+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2822571,"num_objects":345,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11535,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1717}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11615","reported_seq":15658,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:15.775247+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:15.775247+0000","last_peered":"2024-09-15T03:17:15.775247+0000","last_clean":"2024-09-15T03:17:15.775247+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:15.775247+0000","last_undegraded":"2024-09-15T03:17:15.775247+0000","last_fullsized":"2024-09-15T03:17:15.775247+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2325305,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11831","reported_seq":16007,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:15.593399+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:15.593399+0000","last_peered":"2024-09-15T03:17:15.593399+0000","last_clean":"2024-09-15T03:17:15.593399+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:15.593399+0000","last_undegraded":"2024-09-15T03:17:15.593399+0000","last_fullsized":"2024-09-15T03:17:15.593399+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2472761,"num_objects":302,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11358","reported_seq":11460,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:11.868380+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:11.868380+0000","last_peered":"2024-09-15T03:17:11.868380+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:11.868380+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2651762,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11356,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1689}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11610","reported_seq":11719,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.665186+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:16.665186+0000","last_peered":"2024-09-15T03:17:16.665186+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:16.665186+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2233970,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11608,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1704}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11731","reported_seq":11846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.867197+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:16.867197+0000","last_peered":"2024-09-15T03:17:16.867197+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:16.867197+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2274930,"num_objects":278,"num_object_clones":0,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11729,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1781}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":39598676,"num_objects":4839,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185868,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7638,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77590528,"data_stored":77590528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":940992,"kb_used_data":21752,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93430848,"statfs":{"total":96636764160,"available":95673188352,"internally_reserved":0,"allocated":22274048,"data_stored":22090227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,5,5,4,1,3,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.34100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.53200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":945560,"kb_used_data":18448,"kb_used_omap":1,"kb_used_meta":927102,"kb_avail":93426280,"statfs":{"total":96636764160,"available":95668510720,"internally_reserved":0,"allocated":18890752,"data_stored":18710077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949352792},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":5.9249999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.434}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.379},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":5.8810000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614028,"kb_used_data":19456,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93757812,"statfs":{"total":96636764160,"available":96007999488,"internally_reserved":0,"allocated":19922944,"data_stored":19739123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.36499999999999999},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":1.794}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.217}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":1.833},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.29599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933248,"kb_used_data":18168,"kb_used_omap":4,"kb_used_meta":915067,"kb_avail":93438592,"statfs":{"total":96636764160,"available":95681118208,"internally_reserved":0,"allocated":18604032,"data_stored":18420211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937029185},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,0,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.372},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.48999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":7.9699999999999998},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.47999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.311},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18223104,"data_stored":18223104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19193856,"data_stored":19193856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18300928,"data_stored":18300928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":21872640,"data_stored":21872640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:18.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:19.793 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:19.815 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:19.873 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-15T03:17:18.975639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36780628,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186212,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7638,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160539,"ondisk_log_size":160539,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3434404,"kb_used_data":74240,"kb_used_omap":14,"kb_used_meta":3360113,"kb_avail":374052956,"statfs":{"total":386547056640,"available":383030226944,"internally_reserved":0,"allocated":76021760,"data_stored":75289622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3440755739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,6,9,16,7,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14663680,"num_objects":-1790,"num_object_clones":0,"num_object_copies":-7160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1790,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006881"},"pg_stats":[{"pgid":"1.f","version":"37'11635","reported_seq":11703,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.845574+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:16.845574+0000","last_peered":"2024-09-15T03:17:16.845574+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:16.845574+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2252800,"num_objects":275,"num_object_clones":0,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11164","reported_seq":14065,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.861985+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:16.861985+0000","last_peered":"2024-09-15T03:17:16.861985+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:16.861985+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2143858,"num_objects":262,"num_object_clones":0,"num_object_copies":1048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11162,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11928","reported_seq":12038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.740499+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:16.740499+0000","last_peered":"2024-09-15T03:17:16.740499+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:16.740499+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2390865,"num_objects":293,"num_object_clones":0,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11927,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1798}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11340","reported_seq":15313,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:15.821734+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:15.821734+0000","last_peered":"2024-09-15T03:17:15.821734+0000","last_clean":"2024-09-15T03:17:15.821734+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:15.821734+0000","last_undegraded":"2024-09-15T03:17:15.821734+0000","last_fullsized":"2024-09-15T03:17:15.821734+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2356850,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11338,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11807","reported_seq":11914,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.941203+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:16.941203+0000","last_peered":"2024-09-15T03:17:16.941203+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:16.941203+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2242162,"num_objects":274,"num_object_clones":0,"num_object_copies":1096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11805,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11733","reported_seq":11819,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.876696+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:16.876696+0000","last_peered":"2024-09-15T03:17:16.876696+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:16.876696+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2170880,"num_objects":265,"num_object_clones":0,"num_object_copies":1060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1773}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11667","reported_seq":11790,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.708220+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:16.708220+0000","last_peered":"2024-09-15T03:17:16.708220+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:16.708220+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2381426,"num_objects":291,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11665,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1735}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11812","reported_seq":11931,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.851635+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:16.851635+0000","last_peered":"2024-09-15T03:17:16.851635+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:16.851635+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2233970,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1783}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11670","reported_seq":11763,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.710137+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:16.710137+0000","last_peered":"2024-09-15T03:17:16.710137+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:16.710137+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2452708,"num_objects":300,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11667,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1727}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11684","reported_seq":11805,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.691782+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:16.691782+0000","last_peered":"2024-09-15T03:17:16.691782+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:16.691782+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2324082,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11601","reported_seq":11697,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.672230+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:16.672230+0000","last_peered":"2024-09-15T03:17:16.672230+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:16.672230+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2306475,"num_objects":282,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11598,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1780}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11615","reported_seq":15658,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:15.775247+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:15.775247+0000","last_peered":"2024-09-15T03:17:15.775247+0000","last_clean":"2024-09-15T03:17:15.775247+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:15.775247+0000","last_undegraded":"2024-09-15T03:17:15.775247+0000","last_fullsized":"2024-09-15T03:17:15.775247+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2325305,"num_objects":284,"num_object_clones":0,"num_object_copies":1136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11614,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11831","reported_seq":16007,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:15.593399+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:15.593399+0000","last_peered":"2024-09-15T03:17:15.593399+0000","last_clean":"2024-09-15T03:17:15.593399+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:15.593399+0000","last_undegraded":"2024-09-15T03:17:15.593399+0000","last_fullsized":"2024-09-15T03:17:15.593399+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2472761,"num_objects":302,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11411","reported_seq":11513,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.835664+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:16.835664+0000","last_peered":"2024-09-15T03:17:16.835664+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:16.835664+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2217586,"num_objects":271,"num_object_clones":0,"num_object_copies":1084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11409,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1742}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11610","reported_seq":11719,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.665186+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:16.665186+0000","last_peered":"2024-09-15T03:17:16.665186+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:16.665186+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2233970,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11608,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1704}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11731","reported_seq":11846,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:16.867197+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:16.867197+0000","last_peered":"2024-09-15T03:17:16.867197+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:16.867197+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2274930,"num_objects":278,"num_object_clones":0,"num_object_copies":1112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11729,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1781}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":36780628,"num_objects":4495,"num_object_clones":0,"num_object_copies":17980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25046,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":186212,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7638,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73908224,"data_stored":73908224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160539,"ondisk_log_size":160539,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":941568,"kb_used_data":18168,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93430272,"statfs":{"total":96636764160,"available":95672598528,"internally_reserved":0,"allocated":18604032,"data_stored":18420211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,4,5,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":2.8239999999999998},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":4.008}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":3.1320000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":2.863}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":3.0979999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":18.527999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":945560,"kb_used_data":18448,"kb_used_omap":1,"kb_used_meta":927102,"kb_avail":93426280,"statfs":{"total":96636764160,"available":95668510720,"internally_reserved":0,"allocated":18890752,"data_stored":18710077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949352792},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":5.9249999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.40999999999999998},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.434}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.379},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":5.8810000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614028,"kb_used_data":19456,"kb_used_omap":4,"kb_used_meta":594555,"kb_avail":93757812,"statfs":{"total":96636764160,"available":96007999488,"internally_reserved":0,"allocated":19922944,"data_stored":19739123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":608824897},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,1,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.36499999999999999},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":1.794}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.217}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":1.833},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.29599999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933248,"kb_used_data":18168,"kb_used_omap":4,"kb_used_meta":915067,"kb_avail":93438592,"statfs":{"total":96636764160,"available":95681118208,"internally_reserved":0,"allocated":18604032,"data_stored":18420211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937029185},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,0,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.372},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.48999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":7.9699999999999998},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.47999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.311},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18223104,"data_stored":18223104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19193856,"data_stored":19193856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18300928,"data_stored":18300928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18190336,"data_stored":18190336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:19.874 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11635", 'reported_seq': 11703, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.845574+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:16.845574+0000', 'last_peered': '2024-09-15T03:17:16.845574+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:16.845574+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2252800, 'num_objects': 275, 'num_object_clones': 0, 'num_object_copies': 1100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 275, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11635, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1691}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11164", 'reported_seq': 14065, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.861985+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:17:16.861985+0000', 'last_peered': '2024-09-15T03:17:16.861985+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:17:16.861985+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2143858, 'num_objects': 262, 'num_object_clones': 0, 'num_object_copies': 1048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 584, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11162, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1418, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -322}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11928", 'reported_seq': 12038, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.740499+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:16.740499+0000', 'last_peered': '2024-09-15T03:17:16.740499+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:16.740499+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2390865, 'num_objects': 293, 'num_object_clones': 0, 'num_object_copies': 1172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 293, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11927, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1798}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11807", 'reported_seq': 11914, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.941203+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:16.941203+0000', 'last_peered': '2024-09-15T03:17:16.941203+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:16.941203+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2242162, 'num_objects': 274, 'num_object_clones': 0, 'num_object_copies': 1096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11805, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1810}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11733", 'reported_seq': 11819, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.876696+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:16.876696+0000', 'last_peered': '2024-09-15T03:17:16.876696+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:16.876696+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2170880, 'num_objects': 265, 'num_object_clones': 0, 'num_object_copies': 1060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11733, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1773}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:19.875 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11667", 'reported_seq': 11790, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.708220+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:16.708220+0000', 'last_peered': '2024-09-15T03:17:16.708220+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:16.708220+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2381426, 'num_objects': 291, 'num_object_clones': 0, 'num_object_copies': 1164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11665, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1735}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11812", 'reported_seq': 11931, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.851635+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:16.851635+0000', 'last_peered': '2024-09-15T03:17:16.851635+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:16.851635+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2233970, 'num_objects': 273, 'num_object_clones': 0, 'num_object_copies': 1092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11810, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11670", 'reported_seq': 11763, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.710137+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:17:16.710137+0000', 'last_peered': '2024-09-15T03:17:16.710137+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:17:16.710137+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2452708, 'num_objects': 300, 'num_object_clones': 0, 'num_object_copies': 1200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11667, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1727}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11684", 'reported_seq': 11805, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.691782+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:16.691782+0000', 'last_peered': '2024-09-15T03:17:16.691782+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:16.691782+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2324082, 'num_objects': 284, 'num_object_clones': 0, 'num_object_copies': 1136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11682, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1758}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11601", 'reported_seq': 11697, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.672230+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:16.672230+0000', 'last_peered': '2024-09-15T03:17:16.672230+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:16.672230+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2306475, 'num_objects': 282, 'num_object_clones': 0, 'num_object_copies': 1128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 282, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11598, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1780}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:17:19.876 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11411", 'reported_seq': 11513, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.835664+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:16.835664+0000', 'last_peered': '2024-09-15T03:17:16.835664+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:16.835664+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2217586, 'num_objects': 271, 'num_object_clones': 0, 'num_object_copies': 1084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11409, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1742}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:19.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:17:19.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11610", 'reported_seq': 11719, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.665186+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:16.665186+0000', 'last_peered': '2024-09-15T03:17:16.665186+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:16.665186+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2233970, 'num_objects': 273, 'num_object_clones': 0, 'num_object_copies': 1092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11608, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1704}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:19.877 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:17:19.877 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11731", 'reported_seq': 11846, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:16.867197+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:16.867197+0000', 'last_peered': '2024-09-15T03:17:16.867197+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:16.867197+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2274930, 'num_objects': 278, 'num_object_clones': 0, 'num_object_copies': 1112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11729, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1781}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:19.877 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:17:21.873 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:21.873 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25046/17980 objects degraded (139.299%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":764,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4495,"data_bytes":36780628,"bytes_used":3516829696,"bytes_avail":383030226944,"bytes_total":386547056640,"degraded_objects":25046,"degraded_total":17980,"degraded_ratio":1.3929922135706341,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":149},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:17:21.941 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:17:21.941 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:23.750 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:23.836 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:23.843 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-15T03:17:22.980407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29661780,"num_objects":3626,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187081,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7685,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161008,"ondisk_log_size":161008,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3420608,"kb_used_data":60448,"kb_used_omap":14,"kb_used_meta":3360113,"kb_avail":374066752,"statfs":{"total":386547056640,"available":383044354048,"internally_reserved":0,"allocated":61898752,"data_stored":61166614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3440755739},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,5,10,9,8,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20447232,"num_objects":-2496,"num_object_clones":0,"num_object_copies":-9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010665"},"pg_stats":[{"pgid":"1.f","version":"37'11689","reported_seq":11757,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.851330+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:21.851330+0000","last_peered":"2024-09-15T03:17:21.851330+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:21.851330+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1810432,"num_objects":221,"num_object_clones":0,"num_object_copies":884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11689,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11224","reported_seq":14219,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:22.012468+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:22.012468+0000","last_peered":"2024-09-15T03:17:22.012468+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:22.012468+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1652338,"num_objects":202,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11988","reported_seq":12098,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.768316+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:21.768316+0000","last_peered":"2024-09-15T03:17:21.768316+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:21.768316+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1899345,"num_objects":233,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11987,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1858}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11384","reported_seq":15358,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:20.568575+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:20.568575+0000","last_peered":"2024-09-15T03:17:20.568575+0000","last_clean":"2024-09-15T03:17:20.568575+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:20.568575+0000","last_undegraded":"2024-09-15T03:17:20.568575+0000","last_fullsized":"2024-09-15T03:17:20.568575+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1996402,"num_objects":244,"num_object_clones":0,"num_object_copies":976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11382,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11849","reported_seq":11957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.698447+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:21.698447+0000","last_peered":"2024-09-15T03:17:21.698447+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:21.698447+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1898098,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11795","reported_seq":11881,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.426270+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:21.426270+0000","last_peered":"2024-09-15T03:17:21.426270+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:21.426270+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1662976,"num_objects":203,"num_object_clones":0,"num_object_copies":812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11730","reported_seq":11853,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.961172+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:21.961172+0000","last_peered":"2024-09-15T03:17:21.961172+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:21.961172+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1865330,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1798}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11867","reported_seq":11986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.687564+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:21.687564+0000","last_peered":"2024-09-15T03:17:21.687564+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:21.687564+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1783410,"num_objects":218,"num_object_clones":0,"num_object_copies":872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11865,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11718","reported_seq":11811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:22.021929+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:22.021929+0000","last_peered":"2024-09-15T03:17:22.021929+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:22.021929+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2059492,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11751","reported_seq":11869,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.852106+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:21.852106+0000","last_peered":"2024-09-15T03:17:21.852106+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:21.852106+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1775218,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1825}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11652","reported_seq":11748,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.851630+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:21.851630+0000","last_peered":"2024-09-15T03:17:21.851630+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:21.851630+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1888683,"num_objects":231,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11675","reported_seq":15718,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:20.953133+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:20.953133+0000","last_peered":"2024-09-15T03:17:20.953133+0000","last_clean":"2024-09-15T03:17:20.953133+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:20.953133+0000","last_undegraded":"2024-09-15T03:17:20.953133+0000","last_fullsized":"2024-09-15T03:17:20.953133+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1833785,"num_objects":224,"num_object_clones":0,"num_object_copies":896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11674,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11884","reported_seq":16060,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:20.819258+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:20.819258+0000","last_peered":"2024-09-15T03:17:20.819258+0000","last_clean":"2024-09-15T03:17:20.819258+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:20.819258+0000","last_undegraded":"2024-09-15T03:17:20.819258+0000","last_fullsized":"2024-09-15T03:17:20.819258+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2038585,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11456","reported_seq":11558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.903087+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:21.903087+0000","last_peered":"2024-09-15T03:17:21.903087+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:21.903087+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1848946,"num_objects":226,"num_object_clones":0,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11667","reported_seq":11776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.914643+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:21.914643+0000","last_peered":"2024-09-15T03:17:21.914643+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:21.914643+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1767026,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11665,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1761}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11779","reported_seq":11894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.873456+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:21.873456+0000","last_peered":"2024-09-15T03:17:21.873456+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:21.873456+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1881714,"num_objects":230,"num_object_clones":0,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":29661780,"num_objects":3626,"num_object_clones":0,"num_object_copies":14504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187081,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7685,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":59543552,"data_stored":59543552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161008,"ondisk_log_size":161008,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":942280,"kb_used_data":14720,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93429560,"statfs":{"total":96636764160,"available":95671869440,"internally_reserved":0,"allocated":15073280,"data_stored":14889459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.311},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.217}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.38200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":929700,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93442140,"statfs":{"total":96636764160,"available":95684751360,"internally_reserved":0,"allocated":15429632,"data_stored":15248957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.372}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614644,"kb_used_data":15916,"kb_used_omap":4,"kb_used_meta":598715,"kb_avail":93757196,"statfs":{"total":96636764160,"available":96007368704,"internally_reserved":0,"allocated":16297984,"data_stored":16114163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":613084737},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,5,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.20599999999999999},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.32800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.1619999999999999},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.371}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":4.1909999999999998},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.2249999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933984,"kb_used_data":14744,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93437856,"statfs":{"total":96636764160,"available":95680364544,"internally_reserved":0,"allocated":15097856,"data_stored":14914035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.433},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.54500000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.34899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14643200,"data_stored":14643200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15577088,"data_stored":15577088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14692352,"data_stored":14692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14630912,"data_stored":14630912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:26.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:28.396 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:28.404 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:28.415 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-15T03:17:26.981396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28375636,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187238,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7685,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160865,"ondisk_log_size":160865,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3739888,"kb_used_data":57104,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373747472,"statfs":{"total":386547056640,"available":382717411328,"internally_reserved":0,"allocated":58474496,"data_stored":57742358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,4,11,7,11,11,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15892480,"num_objects":-1940,"num_object_clones":0,"num_object_copies":-7760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1940,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008470"},"pg_stats":[{"pgid":"1.f","version":"37'11689","reported_seq":11757,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.851330+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:21.851330+0000","last_peered":"2024-09-15T03:17:21.851330+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:21.851330+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1810432,"num_objects":221,"num_object_clones":0,"num_object_copies":884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11689,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11224","reported_seq":14219,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:22.012468+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:22.012468+0000","last_peered":"2024-09-15T03:17:22.012468+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:22.012468+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1652338,"num_objects":202,"num_object_clones":0,"num_object_copies":808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11222,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1465,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'11988","reported_seq":12098,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.768316+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:21.768316+0000","last_peered":"2024-09-15T03:17:21.768316+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:21.768316+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1899345,"num_objects":233,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":233,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11987,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1858}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11433","reported_seq":15407,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:25.979873+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:25.979873+0000","last_peered":"2024-09-15T03:17:25.979873+0000","last_clean":"2024-09-15T03:17:25.979873+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:25.979873+0000","last_undegraded":"2024-09-15T03:17:25.979873+0000","last_fullsized":"2024-09-15T03:17:25.979873+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594994,"num_objects":195,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11849","reported_seq":11957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.698447+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:21.698447+0000","last_peered":"2024-09-15T03:17:21.698447+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:21.698447+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1898098,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11847,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11795","reported_seq":11881,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.426270+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:21.426270+0000","last_peered":"2024-09-15T03:17:21.426270+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:21.426270+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1662976,"num_objects":203,"num_object_clones":0,"num_object_copies":812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1835}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11730","reported_seq":11853,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.961172+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:21.961172+0000","last_peered":"2024-09-15T03:17:21.961172+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:21.961172+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1865330,"num_objects":228,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1798}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11867","reported_seq":11986,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.687564+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:21.687564+0000","last_peered":"2024-09-15T03:17:21.687564+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:21.687564+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1783410,"num_objects":218,"num_object_clones":0,"num_object_copies":872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11865,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1838}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11718","reported_seq":11811,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:22.021929+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:22.021929+0000","last_peered":"2024-09-15T03:17:22.021929+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:22.021929+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2059492,"num_objects":252,"num_object_clones":0,"num_object_copies":1008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11751","reported_seq":11869,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.852106+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:21.852106+0000","last_peered":"2024-09-15T03:17:21.852106+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:21.852106+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1775218,"num_objects":217,"num_object_clones":0,"num_object_copies":868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11749,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1825}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11652","reported_seq":11748,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.851630+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:21.851630+0000","last_peered":"2024-09-15T03:17:21.851630+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:21.851630+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1888683,"num_objects":231,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11649,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11728","reported_seq":15771,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:25.711086+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:25.711086+0000","last_peered":"2024-09-15T03:17:25.711086+0000","last_clean":"2024-09-15T03:17:25.711086+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:25.711086+0000","last_undegraded":"2024-09-15T03:17:25.711086+0000","last_fullsized":"2024-09-15T03:17:25.711086+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1399609,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11939","reported_seq":16115,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:26.084044+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:26.084044+0000","last_peered":"2024-09-15T03:17:26.084044+0000","last_clean":"2024-09-15T03:17:26.084044+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:26.084044+0000","last_undegraded":"2024-09-15T03:17:26.084044+0000","last_fullsized":"2024-09-15T03:17:26.084044+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1588025,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11938,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11456","reported_seq":11558,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.903087+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:21.903087+0000","last_peered":"2024-09-15T03:17:21.903087+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:21.903087+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1848946,"num_objects":226,"num_object_clones":0,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11454,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11667","reported_seq":11776,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.914643+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:21.914643+0000","last_peered":"2024-09-15T03:17:21.914643+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:21.914643+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1767026,"num_objects":216,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11665,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1761}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11779","reported_seq":11894,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:21.873456+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:21.873456+0000","last_peered":"2024-09-15T03:17:21.873456+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:21.873456+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1881714,"num_objects":230,"num_object_clones":0,"num_object_copies":920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":28375636,"num_objects":3469,"num_object_clones":0,"num_object_copies":13876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3469,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187238,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7685,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":55980032,"data_stored":55980032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160865,"ondisk_log_size":160865,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":942280,"kb_used_data":14720,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93429560,"statfs":{"total":96636764160,"available":95671869440,"internally_reserved":0,"allocated":15073280,"data_stored":14889459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.311},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.217}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":1.2529999999999999},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.38200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":929700,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93442140,"statfs":{"total":96636764160,"available":95684751360,"internally_reserved":0,"allocated":15429632,"data_stored":15248957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:16:21 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.7370000000000001,"15min":1.7749999999999999},"min":{"1min":0.22700000000000001,"5min":0.20499999999999999,"15min":0.19900000000000001},"max":{"1min":10.968,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":2.488,"5min":2.7810000000000001,"15min":1.831},"min":{"1min":0.29299999999999998,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":11.002000000000001,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.372}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46500000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933924,"kb_used_data":12572,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93437916,"statfs":{"total":96636764160,"available":95680425984,"internally_reserved":0,"allocated":12873728,"data_stored":12689907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,3,6,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.38200000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.434},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.33100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933984,"kb_used_data":14744,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93437856,"statfs":{"total":96636764160,"available":95680364544,"internally_reserved":0,"allocated":15097856,"data_stored":14914035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.433},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":0.54500000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.51000000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.34899999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14643200,"data_stored":14643200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12013568,"data_stored":12013568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14692352,"data_stored":14692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14630912,"data_stored":14630912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:28.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:30.290 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:30.300 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:30.397 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-15T03:17:28.981908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22510164,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187954,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7748,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160581,"ondisk_log_size":160581,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3725296,"kb_used_data":46668,"kb_used_omap":14,"kb_used_meta":3678577,"kb_avail":373762064,"statfs":{"total":386547056640,"available":382732353536,"internally_reserved":0,"allocated":47788032,"data_stored":47055894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3766862875},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,1,12,11,11,13,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17088512,"num_objects":-2086,"num_object_clones":0,"num_object_copies":-8344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006776"},"pg_stats":[{"pgid":"1.f","version":"37'11746","reported_seq":11814,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.882576+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:26.882576+0000","last_peered":"2024-09-15T03:17:26.882576+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:26.882576+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1343488,"num_objects":164,"num_object_clones":0,"num_object_copies":656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11746,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1802}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11269","reported_seq":14390,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.983385+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:26.983385+0000","last_peered":"2024-09-15T03:17:26.983385+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:26.983385+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1283698,"num_objects":157,"num_object_clones":0,"num_object_copies":628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11267,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1528,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-317}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12049","reported_seq":12157,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.964946+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:26.964946+0000","last_peered":"2024-09-15T03:17:26.964946+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:26.964946+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1399633,"num_objects":172,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12048,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11433","reported_seq":15407,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:25.979873+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:25.979873+0000","last_peered":"2024-09-15T03:17:25.979873+0000","last_clean":"2024-09-15T03:17:25.979873+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:25.979873+0000","last_undegraded":"2024-09-15T03:17:25.979873+0000","last_fullsized":"2024-09-15T03:17:25.979873+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594994,"num_objects":195,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11901","reported_seq":12009,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.948541+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:26.948541+0000","last_peered":"2024-09-15T03:17:26.948541+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:26.948541+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1472114,"num_objects":180,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11899,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1904}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11850","reported_seq":11936,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:27.089465+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:27.089465+0000","last_peered":"2024-09-15T03:17:27.089465+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:27.089465+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1212416,"num_objects":148,"num_object_clones":0,"num_object_copies":592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11850,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11778","reported_seq":11901,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.874237+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:26.874237+0000","last_peered":"2024-09-15T03:17:26.874237+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:26.874237+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1472114,"num_objects":180,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1846}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11927","reported_seq":12046,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.860968+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:26.860968+0000","last_peered":"2024-09-15T03:17:26.860968+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:26.860968+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1291890,"num_objects":158,"num_object_clones":0,"num_object_copies":632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11925,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11773","reported_seq":11866,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:27.029405+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:27.029405+0000","last_peered":"2024-09-15T03:17:27.029405+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:27.029405+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1608932,"num_objects":197,"num_object_clones":0,"num_object_copies":788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11770,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11806","reported_seq":11927,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.612046+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:26.612046+0000","last_peered":"2024-09-15T03:17:26.612046+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:26.612046+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1324658,"num_objects":162,"num_object_clones":0,"num_object_copies":648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11804,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11705","reported_seq":11801,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.996584+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:26.996584+0000","last_peered":"2024-09-15T03:17:26.996584+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:26.996584+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1454507,"num_objects":178,"num_object_clones":0,"num_object_copies":712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11702,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11728","reported_seq":15771,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:25.711086+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:25.711086+0000","last_peered":"2024-09-15T03:17:25.711086+0000","last_clean":"2024-09-15T03:17:25.711086+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:25.711086+0000","last_undegraded":"2024-09-15T03:17:25.711086+0000","last_fullsized":"2024-09-15T03:17:25.711086+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1399609,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11939","reported_seq":16115,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:26.084044+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:26.084044+0000","last_peered":"2024-09-15T03:17:26.084044+0000","last_clean":"2024-09-15T03:17:26.084044+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:26.084044+0000","last_undegraded":"2024-09-15T03:17:26.084044+0000","last_fullsized":"2024-09-15T03:17:26.084044+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1588025,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11938,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11511","reported_seq":11612,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.961429+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:26.961429+0000","last_peered":"2024-09-15T03:17:26.961429+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:26.961429+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1398386,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11509,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1842}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11729","reported_seq":11838,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:27.031421+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:27.031421+0000","last_peered":"2024-09-15T03:17:27.031421+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:27.031421+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1259122,"num_objects":154,"num_object_clones":0,"num_object_copies":616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1823}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11837","reported_seq":11952,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:26.939905+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:26.939905+0000","last_peered":"2024-09-15T03:17:26.939905+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:26.939905+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1406578,"num_objects":172,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11835,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1887}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":22510164,"num_objects":2753,"num_object_clones":0,"num_object_copies":11012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24936,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187954,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7748,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45244416,"data_stored":45244416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160581,"ondisk_log_size":160581,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926360,"kb_used_data":11280,"kb_used_omap":4,"kb_used_meta":915067,"kb_avail":93445480,"statfs":{"total":96636764160,"available":95688171520,"internally_reserved":0,"allocated":11550720,"data_stored":11366899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937029185},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,6,3,4,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.373},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.23599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":2.1400000000000001},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":2.1800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.40300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930452,"kb_used_data":11660,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93441388,"statfs":{"total":96636764160,"available":95683981312,"internally_reserved":0,"allocated":11939840,"data_stored":11759165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.308},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.72499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.1920000000000002},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.2309999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.34599999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":933924,"kb_used_data":12572,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93437916,"statfs":{"total":96636764160,"available":95680425984,"internally_reserved":0,"allocated":12873728,"data_stored":12689907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,3,6,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.38200000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.434},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.33100000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934560,"kb_used_data":11156,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93437280,"statfs":{"total":96636764160,"available":95679774720,"internally_reserved":0,"allocated":11423744,"data_stored":11239923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:16:29 2024","interfaces":[{"interface":"back","average":{"1min":2.2360000000000002,"5min":2.218,"15min":1.5700000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":22.384,"5min":22.384,"15min":22.384},"last":0.53800000000000003},{"interface":"front","average":{"1min":4.218,"5min":3.2909999999999999,"15min":2.5209999999999999},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":30.433,"5min":30.433,"15min":30.433},"last":14.662000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":14.718999999999999},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.27900000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.32800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11079680,"data_stored":11079680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12013568,"data_stored":12013568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11079680,"data_stored":11079680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":11071488,"data_stored":11071488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:30.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:17:30.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11746", 'reported_seq': 11814, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.882576+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:26.882576+0000', 'last_peered': '2024-09-15T03:17:26.882576+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:26.882576+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1343488, 'num_objects': 164, 'num_object_clones': 0, 'num_object_copies': 656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11746, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1802}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:30.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:17:30.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11269", 'reported_seq': 14390, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.983385+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:17:26.983385+0000', 'last_peered': '2024-09-15T03:17:26.983385+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:17:26.983385+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1283698, 'num_objects': 157, 'num_object_clones': 0, 'num_object_copies': 628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 474, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 157, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11267, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1528, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -317}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:30.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:17:30.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12049", 'reported_seq': 12157, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.964946+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:26.964946+0000', 'last_peered': '2024-09-15T03:17:26.964946+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:26.964946+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1399633, 'num_objects': 172, 'num_object_clones': 0, 'num_object_copies': 688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 172, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12048, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1919}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:30.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11901", 'reported_seq': 12009, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.948541+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:26.948541+0000', 'last_peered': '2024-09-15T03:17:26.948541+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:26.948541+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1472114, 'num_objects': 180, 'num_object_clones': 0, 'num_object_copies': 720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11899, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1904}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11850", 'reported_seq': 11936, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:27.089465+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:27.089465+0000', 'last_peered': '2024-09-15T03:17:27.089465+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:27.089465+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1212416, 'num_objects': 148, 'num_object_clones': 0, 'num_object_copies': 592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11850, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1890}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11778", 'reported_seq': 11901, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.874237+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:26.874237+0000', 'last_peered': '2024-09-15T03:17:26.874237+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:26.874237+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1472114, 'num_objects': 180, 'num_object_clones': 0, 'num_object_copies': 720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11776, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1846}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11927", 'reported_seq': 12046, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.860968+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:26.860968+0000', 'last_peered': '2024-09-15T03:17:26.860968+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:26.860968+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1291890, 'num_objects': 158, 'num_object_clones': 0, 'num_object_copies': 632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11925, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11773", 'reported_seq': 11866, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:27.029405+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:17:27.029405+0000', 'last_peered': '2024-09-15T03:17:27.029405+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:17:27.029405+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1608932, 'num_objects': 197, 'num_object_clones': 0, 'num_object_copies': 788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 197, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11770, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1830}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:30.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11806", 'reported_seq': 11927, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.612046+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:26.612046+0000', 'last_peered': '2024-09-15T03:17:26.612046+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:26.612046+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1324658, 'num_objects': 162, 'num_object_clones': 0, 'num_object_copies': 648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 162, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11804, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1880}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11705", 'reported_seq': 11801, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.996584+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:26.996584+0000', 'last_peered': '2024-09-15T03:17:26.996584+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:26.996584+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1454507, 'num_objects': 178, 'num_object_clones': 0, 'num_object_copies': 712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11702, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1884}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11511", 'reported_seq': 11612, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.961429+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:26.961429+0000', 'last_peered': '2024-09-15T03:17:26.961429+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:26.961429+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1398386, 'num_objects': 171, 'num_object_clones': 0, 'num_object_copies': 684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11509, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1842}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11729", 'reported_seq': 11838, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:27.031421+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:27.031421+0000', 'last_peered': '2024-09-15T03:17:27.031421+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:27.031421+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1259122, 'num_objects': 154, 'num_object_clones': 0, 'num_object_copies': 616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11727, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1823}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:17:30.401 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11837", 'reported_seq': 11952, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:26.939905+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:26.939905+0000', 'last_peered': '2024-09-15T03:17:26.939905+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:26.939905+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1406578, 'num_objects': 172, 'num_object_clones': 0, 'num_object_copies': 688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11835, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:30.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:17:32.647 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:32.647 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24936/11012 objects degraded (226.444%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":775,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2753,"data_bytes":22510164,"bytes_used":3814703104,"bytes_avail":382732353536,"bytes_total":386547056640,"degraded_objects":24936,"degraded_total":11012,"degraded_ratio":2.2644387940428623,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":110,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":145},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:17:32.716 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:17:32.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:34.510 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:34.540 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:34.550 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-15T03:17:32.982794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":15243860,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188841,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7799,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161068,"ondisk_log_size":161068,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3727756,"kb_used_data":32484,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373759604,"statfs":{"total":386547056640,"available":382729834496,"internally_reserved":0,"allocated":33263616,"data_stored":32531478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,5,11,6,15,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21536768,"num_objects":-2629,"num_object_clones":0,"num_object_copies":-10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006656"},"pg_stats":[{"pgid":"1.f","version":"37'11795","reported_seq":11863,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:31.954436+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:31.954436+0000","last_peered":"2024-09-15T03:17:31.954436+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:31.954436+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":942080,"num_objects":115,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11325","reported_seq":14547,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:32.145208+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:32.145208+0000","last_peered":"2024-09-15T03:17:32.145208+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:32.145208+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":824946,"num_objects":101,"num_object_clones":0,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11323,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12107","reported_seq":12217,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:31.722940+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:31.722940+0000","last_peered":"2024-09-15T03:17:31.722940+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:31.722940+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":924497,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12106,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1977}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11489","reported_seq":15463,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:30.992744+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:30.992744+0000","last_peered":"2024-09-15T03:17:30.992744+0000","last_clean":"2024-09-15T03:17:30.992744+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:30.992744+0000","last_undegraded":"2024-09-15T03:17:30.992744+0000","last_fullsized":"2024-09-15T03:17:30.992744+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1136242,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11487,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'11967","reported_seq":12075,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:31.998156+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:31.998156+0000","last_peered":"2024-09-15T03:17:31.998156+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:31.998156+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":931442,"num_objects":114,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11965,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11892","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:32.071801+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:32.071801+0000","last_peered":"2024-09-15T03:17:32.071801+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:32.071801+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":868352,"num_objects":106,"num_object_clones":0,"num_object_copies":424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11892,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1932}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11839","reported_seq":11962,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:32.028536+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:32.028536+0000","last_peered":"2024-09-15T03:17:32.028536+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:32.028536+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":972402,"num_objects":119,"num_object_clones":0,"num_object_copies":476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11837,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1907}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11984","reported_seq":12101,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:32.127042+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:32.127042+0000","last_peered":"2024-09-15T03:17:32.127042+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:32.127042+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":824946,"num_objects":101,"num_object_clones":0,"num_object_copies":404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11982,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1955}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11841","reported_seq":11934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:31.897002+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:31.897002+0000","last_peered":"2024-09-15T03:17:31.897002+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:31.897002+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1051876,"num_objects":129,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11838,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1898}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11859","reported_seq":11980,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:31.659059+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:31.659059+0000","last_peered":"2024-09-15T03:17:31.659059+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:31.659059+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":890482,"num_objects":109,"num_object_clones":0,"num_object_copies":436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11857,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1933}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11772","reported_seq":11868,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:32.141673+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:32.141673+0000","last_peered":"2024-09-15T03:17:32.141673+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:32.141673+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":905643,"num_objects":111,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11769,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1951}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11778","reported_seq":15821,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:31.095107+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:31.095107+0000","last_peered":"2024-09-15T03:17:31.095107+0000","last_clean":"2024-09-15T03:17:31.095107+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:31.095107+0000","last_undegraded":"2024-09-15T03:17:31.095107+0000","last_fullsized":"2024-09-15T03:17:31.095107+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":990009,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'11996","reported_seq":16172,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:31.103433+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:31.103433+0000","last_peered":"2024-09-15T03:17:31.103433+0000","last_clean":"2024-09-15T03:17:31.103433+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:31.103433+0000","last_undegraded":"2024-09-15T03:17:31.103433+0000","last_fullsized":"2024-09-15T03:17:31.103433+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1121081,"num_objects":137,"num_object_clones":0,"num_object_copies":548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11995,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11562","reported_seq":11664,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:32.051298+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:32.051298+0000","last_peered":"2024-09-15T03:17:32.051298+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:32.051298+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":980594,"num_objects":120,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11777","reported_seq":11886,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:32.010988+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:32.010988+0000","last_peered":"2024-09-15T03:17:32.010988+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:32.010988+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":865906,"num_objects":106,"num_object_clones":0,"num_object_copies":424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1871}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11885","reported_seq":12000,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:32.131471+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:32.131471+0000","last_peered":"2024-09-15T03:17:32.131471+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:32.131471+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013362,"num_objects":124,"num_object_clones":0,"num_object_copies":496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11883,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1935}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":15243860,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":188841,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7799,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":30658560,"data_stored":30658560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161068,"ondisk_log_size":161068,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926956,"kb_used_data":7716,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93444884,"statfs":{"total":96636764160,"available":95687561216,"internally_reserved":0,"allocated":7901184,"data_stored":7717363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,2,4,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":34.892000000000003},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.91100000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:16:30 2024","interfaces":[{"interface":"back","average":{"1min":5.9240000000000004,"5min":4.2530000000000001,"15min":2.7040000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":27.18,"5min":40.710999999999999,"15min":40.710999999999999},"last":23.995999999999999},{"interface":"front","average":{"1min":4.3369999999999997,"5min":2.605,"15min":1.8460000000000001},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":28.280999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":20.097999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":4.4059999999999997},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":20.236999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931076,"kb_used_data":8116,"kb_used_omap":1,"kb_used_meta":922942,"kb_avail":93440764,"statfs":{"total":96636764160,"available":95683342336,"internally_reserved":0,"allocated":8310784,"data_stored":8130109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":945092952},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.439},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":3.6899999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.35499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":3.5840000000000001},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":3.7370000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934492,"kb_used_data":8980,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93437348,"statfs":{"total":96636764160,"available":95679844352,"internally_reserved":0,"allocated":9195520,"data_stored":9011699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,5,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":6.5259999999999998},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.309}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.26500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.25},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.35199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":935232,"kb_used_data":7672,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93436608,"statfs":{"total":96636764160,"available":95679086592,"internally_reserved":0,"allocated":7856128,"data_stored":7672307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,0,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":20.643999999999998},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":20.353000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.23999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7467008,"data_stored":7467008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8323072,"data_stored":8323072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7475200,"data_stored":7475200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":7393280,"data_stored":7393280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:37.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:39.360 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:39.433 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:39.465 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-15T03:17:38.987720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8092244,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189714,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7857,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160641,"ondisk_log_size":160641,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3717976,"kb_used_data":18528,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373769384,"statfs":{"total":386547056640,"available":382739849216,"internally_reserved":0,"allocated":18972672,"data_stored":18240534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,8,2,18,14,6,7,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20283392,"num_objects":-2476,"num_object_clones":0,"num_object_copies":-9904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006324"},"pg_stats":[{"pgid":"1.f","version":"37'11843","reported_seq":11911,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.120686+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:37.120686+0000","last_peered":"2024-09-15T03:17:37.120686+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:37.120686+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548864,"num_objects":67,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11843,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11371","reported_seq":14710,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.135515+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:37.135515+0000","last_peered":"2024-09-15T03:17:37.135515+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:37.135515+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448114,"num_objects":55,"num_object_clones":0,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11369,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1637,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12161","reported_seq":12271,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.962909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:36.962909+0000","last_peered":"2024-09-15T03:17:36.962909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:36.962909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482129,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12160,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11553","reported_seq":15526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.214794+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:36.214794+0000","last_peered":"2024-09-15T03:17:36.214794+0000","last_clean":"2024-09-15T03:17:36.214794+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:36.214794+0000","last_undegraded":"2024-09-15T03:17:36.214794+0000","last_fullsized":"2024-09-15T03:17:36.214794+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611954,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11551,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12023","reported_seq":12131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.153831+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:37.153831+0000","last_peered":"2024-09-15T03:17:37.153831+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:37.153831+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472690,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12021,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11949","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.883162+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:36.883162+0000","last_peered":"2024-09-15T03:17:36.883162+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:36.883162+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401408,"num_objects":49,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11896","reported_seq":12019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.905908+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:36.905908+0000","last_peered":"2024-09-15T03:17:36.905908+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:36.905908+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505458,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11894,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12033","reported_seq":12152,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.877257+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:36.877257+0000","last_peered":"2024-09-15T03:17:36.877257+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:36.877257+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423538,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11901","reported_seq":11994,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.058091+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:37.058091+0000","last_peered":"2024-09-15T03:17:37.058091+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:37.058091+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":560356,"num_objects":69,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11898,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11909","reported_seq":12030,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.128150+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:37.128150+0000","last_peered":"2024-09-15T03:17:37.128150+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:37.128150+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480882,"num_objects":59,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11907,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11820","reported_seq":11916,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.029953+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:37.029953+0000","last_peered":"2024-09-15T03:17:37.029953+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:37.029953+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512427,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11817,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11831","reported_seq":15874,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.274548+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:36.274548+0000","last_peered":"2024-09-15T03:17:36.274548+0000","last_clean":"2024-09-15T03:17:36.274548+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:36.274548+0000","last_undegraded":"2024-09-15T03:17:36.274548+0000","last_fullsized":"2024-09-15T03:17:36.274548+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":555833,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12049","reported_seq":16225,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.314192+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:36.314192+0000","last_peered":"2024-09-15T03:17:36.314192+0000","last_clean":"2024-09-15T03:17:36.314192+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:36.314192+0000","last_undegraded":"2024-09-15T03:17:36.314192+0000","last_fullsized":"2024-09-15T03:17:36.314192+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":686905,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11621","reported_seq":11723,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.085713+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:37.085713+0000","last_peered":"2024-09-15T03:17:37.085713+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:37.085713+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497266,"num_objects":61,"num_object_clones":0,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11619,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11830","reported_seq":11938,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.110922+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:37.110922+0000","last_peered":"2024-09-15T03:17:37.110922+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:37.110922+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431730,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11951","reported_seq":12063,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.191770+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:37.191770+0000","last_peered":"2024-09-15T03:17:37.191770+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:37.191770+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472690,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":8092244,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189714,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7857,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16211968,"data_stored":16211968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160641,"ondisk_log_size":160641,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927676,"kb_used_data":4276,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93444164,"statfs":{"total":96636764160,"available":95686823936,"internally_reserved":0,"allocated":4378624,"data_stored":4194803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,3,1,2,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.441}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.69499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931720,"kb_used_data":4608,"kb_used_omap":1,"kb_used_meta":927102,"kb_avail":93440120,"statfs":{"total":96636764160,"available":95682682880,"internally_reserved":0,"allocated":4718592,"data_stored":4537917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949352792},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":7.5},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":6.6029999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.3370000000000002},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.4039999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":2.3730000000000002},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":2.319}]}]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":939328,"kb_used_data":5496,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93432512,"statfs":{"total":96636764160,"available":95674892288,"internally_reserved":0,"allocated":5627904,"data_stored":5444083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,1,7,8,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":14.189},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":11.987}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.181},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.22700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":4.1130000000000004},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919252,"kb_used_data":4148,"kb_used_omap":4,"kb_used_meta":915067,"kb_avail":93452588,"statfs":{"total":96636764160,"available":95695450112,"internally_reserved":0,"allocated":4247552,"data_stored":4063731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937029185},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,9,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.27500000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.26200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.46800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.436},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3858432,"data_stored":3858432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4677632,"data_stored":4677632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3870720,"data_stored":3870720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3805184,"data_stored":3805184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:39.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:41.285 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:41.332 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:41.374 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-15T03:17:40.988231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8092244,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189714,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7857,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160641,"ondisk_log_size":160641,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3717976,"kb_used_data":18528,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373769384,"statfs":{"total":386547056640,"available":382739849216,"internally_reserved":0,"allocated":18972672,"data_stored":18240534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,8,2,18,14,6,7,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14417920,"num_objects":-1760,"num_object_clones":0,"num_object_copies":-7040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006322"},"pg_stats":[{"pgid":"1.f","version":"37'11843","reported_seq":11911,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.120686+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:37.120686+0000","last_peered":"2024-09-15T03:17:37.120686+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:37.120686+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548864,"num_objects":67,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11843,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11371","reported_seq":14710,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.135515+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:37.135515+0000","last_peered":"2024-09-15T03:17:37.135515+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:37.135515+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448114,"num_objects":55,"num_object_clones":0,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11369,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1637,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12161","reported_seq":12271,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.962909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:36.962909+0000","last_peered":"2024-09-15T03:17:36.962909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:36.962909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482129,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12160,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11553","reported_seq":15526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.214794+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:36.214794+0000","last_peered":"2024-09-15T03:17:36.214794+0000","last_clean":"2024-09-15T03:17:36.214794+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:36.214794+0000","last_undegraded":"2024-09-15T03:17:36.214794+0000","last_fullsized":"2024-09-15T03:17:36.214794+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611954,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11551,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12023","reported_seq":12131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.153831+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:37.153831+0000","last_peered":"2024-09-15T03:17:37.153831+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:37.153831+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472690,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12021,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11949","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.883162+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:36.883162+0000","last_peered":"2024-09-15T03:17:36.883162+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:36.883162+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401408,"num_objects":49,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11896","reported_seq":12019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.905908+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:36.905908+0000","last_peered":"2024-09-15T03:17:36.905908+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:36.905908+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505458,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11894,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12033","reported_seq":12152,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.877257+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:36.877257+0000","last_peered":"2024-09-15T03:17:36.877257+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:36.877257+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423538,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11901","reported_seq":11994,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.058091+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:37.058091+0000","last_peered":"2024-09-15T03:17:37.058091+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:37.058091+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":560356,"num_objects":69,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11898,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11909","reported_seq":12030,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.128150+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:37.128150+0000","last_peered":"2024-09-15T03:17:37.128150+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:37.128150+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480882,"num_objects":59,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11907,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11820","reported_seq":11916,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.029953+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:37.029953+0000","last_peered":"2024-09-15T03:17:37.029953+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:37.029953+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512427,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11817,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11831","reported_seq":15874,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.274548+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:36.274548+0000","last_peered":"2024-09-15T03:17:36.274548+0000","last_clean":"2024-09-15T03:17:36.274548+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:36.274548+0000","last_undegraded":"2024-09-15T03:17:36.274548+0000","last_fullsized":"2024-09-15T03:17:36.274548+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":555833,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12049","reported_seq":16225,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.314192+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:36.314192+0000","last_peered":"2024-09-15T03:17:36.314192+0000","last_clean":"2024-09-15T03:17:36.314192+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:36.314192+0000","last_undegraded":"2024-09-15T03:17:36.314192+0000","last_fullsized":"2024-09-15T03:17:36.314192+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":686905,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11621","reported_seq":11723,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.085713+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:37.085713+0000","last_peered":"2024-09-15T03:17:37.085713+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:37.085713+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497266,"num_objects":61,"num_object_clones":0,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11619,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11830","reported_seq":11938,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.110922+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:37.110922+0000","last_peered":"2024-09-15T03:17:37.110922+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:37.110922+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431730,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11951","reported_seq":12063,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.191770+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:37.191770+0000","last_peered":"2024-09-15T03:17:37.191770+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:37.191770+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472690,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":8092244,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189714,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7857,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16211968,"data_stored":16211968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160641,"ondisk_log_size":160641,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927676,"kb_used_data":4276,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93444164,"statfs":{"total":96636764160,"available":95686823936,"internally_reserved":0,"allocated":4378624,"data_stored":4194803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,3,1,2,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.441}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.69499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931720,"kb_used_data":4608,"kb_used_omap":1,"kb_used_meta":927102,"kb_avail":93440120,"statfs":{"total":96636764160,"available":95682682880,"internally_reserved":0,"allocated":4718592,"data_stored":4537917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949352792},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":7.5},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":6.6029999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.3370000000000002},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.4039999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":2.3730000000000002},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":2.319}]}]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":939328,"kb_used_data":5496,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93432512,"statfs":{"total":96636764160,"available":95674892288,"internally_reserved":0,"allocated":5627904,"data_stored":5444083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,1,7,8,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":14.189},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":11.987}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.181},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.22700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":4.1130000000000004},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919252,"kb_used_data":4148,"kb_used_omap":4,"kb_used_meta":915067,"kb_avail":93452588,"statfs":{"total":96636764160,"available":95695450112,"internally_reserved":0,"allocated":4247552,"data_stored":4063731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937029185},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,9,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.27500000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.26200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.46800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.436},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3858432,"data_stored":3858432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4677632,"data_stored":4677632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3870720,"data_stored":3870720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3805184,"data_stored":3805184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:41.375 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:17:41.375 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11843", 'reported_seq': 11911, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.120686+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:37.120686+0000', 'last_peered': '2024-09-15T03:17:37.120686+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:37.120686+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 548864, 'num_objects': 67, 'num_object_clones': 0, 'num_object_copies': 268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 67, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11843, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1899}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11371", 'reported_seq': 14710, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.135515+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:17:37.135515+0000', 'last_peered': '2024-09-15T03:17:37.135515+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:17:37.135515+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 448114, 'num_objects': 55, 'num_object_clones': 0, 'num_object_copies': 220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 365, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 55, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11369, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1637, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -310}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12161", 'reported_seq': 12271, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:36.962909+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:36.962909+0000', 'last_peered': '2024-09-15T03:17:36.962909+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:36.962909+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "28'2100", 'ondisk_log_start': "28'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 482129, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 60, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12160, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2031}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12023", 'reported_seq': 12131, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.153831+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:37.153831+0000', 'last_peered': '2024-09-15T03:17:37.153831+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:37.153831+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 472690, 'num_objects': 58, 'num_object_clones': 0, 'num_object_copies': 232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 58, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12021, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2026}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11949", 'reported_seq': 12035, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:36.883162+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:36.883162+0000', 'last_peered': '2024-09-15T03:17:36.883162+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:36.883162+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 401408, 'num_objects': 49, 'num_object_clones': 0, 'num_object_copies': 196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 49, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11949, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1989}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:17:41.376 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11896", 'reported_seq': 12019, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:36.905908+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:36.905908+0000', 'last_peered': '2024-09-15T03:17:36.905908+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:36.905908+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 505458, 'num_objects': 62, 'num_object_clones': 0, 'num_object_copies': 248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 62, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11894, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1964}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12033", 'reported_seq': 12152, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:36.877257+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:36.877257+0000', 'last_peered': '2024-09-15T03:17:36.877257+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:36.877257+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 423538, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 52, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12031, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2004}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11901", 'reported_seq': 11994, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.058091+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:17:37.058091+0000', 'last_peered': '2024-09-15T03:17:37.058091+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:17:37.058091+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 560356, 'num_objects': 69, 'num_object_clones': 0, 'num_object_copies': 276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 69, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11898, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1958}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11909", 'reported_seq': 12030, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.128150+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:37.128150+0000', 'last_peered': '2024-09-15T03:17:37.128150+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:37.128150+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 480882, 'num_objects': 59, 'num_object_clones': 0, 'num_object_copies': 236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 59, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11907, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1983}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11820", 'reported_seq': 11916, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.029953+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:37.029953+0000', 'last_peered': '2024-09-15T03:17:37.029953+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:37.029953+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 512427, 'num_objects': 63, 'num_object_clones': 0, 'num_object_copies': 252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 63, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11817, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1999}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:41.377 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:17:41.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11621", 'reported_seq': 11723, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.085713+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:37.085713+0000', 'last_peered': '2024-09-15T03:17:37.085713+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:37.085713+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 497266, 'num_objects': 61, 'num_object_clones': 0, 'num_object_copies': 244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 61, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11619, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:41.378 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:17:41.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11830", 'reported_seq': 11938, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.110922+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:37.110922+0000', 'last_peered': '2024-09-15T03:17:37.110922+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:37.110922+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 431730, 'num_objects': 53, 'num_object_clones': 0, 'num_object_copies': 212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 53, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11828, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1924}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:41.378 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:17:41.378 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11951", 'reported_seq': 12063, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:37.191770+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:37.191770+0000', 'last_peered': '2024-09-15T03:17:37.191770+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:37.191770+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 472690, 'num_objects': 58, 'num_object_clones': 0, 'num_object_copies': 232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 58, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11949, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2001}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:41.378 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:17:42.502 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Removed 95311 objects 2024-09-15T03:17:42.502 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi060.stdout:Clean up completed and total clean up time :486.044 2024-09-15T03:17:42.664 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:42.664 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24827/3972 objects degraded (625.050%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":785,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":993,"data_bytes":8092244,"bytes_used":3807207424,"bytes_avail":382739849216,"bytes_total":386547056640,"degraded_objects":24827,"degraded_total":3972,"degraded_ratio":6.2505035246727081,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":146},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:17:42.679 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:17:42.679 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:42.968 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:42.970 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:42.979 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-15T03:17:40.988231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8092244,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189714,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7857,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160641,"ondisk_log_size":160641,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3717976,"kb_used_data":18528,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373769384,"statfs":{"total":386547056640,"available":382739849216,"internally_reserved":0,"allocated":18972672,"data_stored":18240534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,8,2,18,14,6,7,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14417920,"num_objects":-1760,"num_object_clones":0,"num_object_copies":-7040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006322"},"pg_stats":[{"pgid":"1.f","version":"37'11843","reported_seq":11911,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.120686+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:37.120686+0000","last_peered":"2024-09-15T03:17:37.120686+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:37.120686+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":548864,"num_objects":67,"num_object_clones":0,"num_object_copies":268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11843,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1899}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11371","reported_seq":14710,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.135515+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:37.135515+0000","last_peered":"2024-09-15T03:17:37.135515+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:37.135515+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448114,"num_objects":55,"num_object_clones":0,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11369,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1637,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-310}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12161","reported_seq":12271,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.962909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:36.962909+0000","last_peered":"2024-09-15T03:17:36.962909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:36.962909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"28'2100","ondisk_log_start":"28'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482129,"num_objects":60,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12160,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2031}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11553","reported_seq":15526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.214794+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:36.214794+0000","last_peered":"2024-09-15T03:17:36.214794+0000","last_clean":"2024-09-15T03:17:36.214794+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:36.214794+0000","last_undegraded":"2024-09-15T03:17:36.214794+0000","last_fullsized":"2024-09-15T03:17:36.214794+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":611954,"num_objects":75,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11551,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12023","reported_seq":12131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.153831+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:37.153831+0000","last_peered":"2024-09-15T03:17:37.153831+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:37.153831+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472690,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12021,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2026}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11949","reported_seq":12035,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.883162+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:36.883162+0000","last_peered":"2024-09-15T03:17:36.883162+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:36.883162+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401408,"num_objects":49,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1989}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11896","reported_seq":12019,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.905908+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:36.905908+0000","last_peered":"2024-09-15T03:17:36.905908+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:36.905908+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505458,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11894,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1964}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12033","reported_seq":12152,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:36.877257+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:36.877257+0000","last_peered":"2024-09-15T03:17:36.877257+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:36.877257+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423538,"num_objects":52,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11901","reported_seq":11994,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.058091+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:37.058091+0000","last_peered":"2024-09-15T03:17:37.058091+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:37.058091+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":560356,"num_objects":69,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11898,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1958}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11909","reported_seq":12030,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.128150+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:37.128150+0000","last_peered":"2024-09-15T03:17:37.128150+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:37.128150+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480882,"num_objects":59,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11907,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11820","reported_seq":11916,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.029953+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:37.029953+0000","last_peered":"2024-09-15T03:17:37.029953+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:37.029953+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512427,"num_objects":63,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11817,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11831","reported_seq":15874,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.274548+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:36.274548+0000","last_peered":"2024-09-15T03:17:36.274548+0000","last_clean":"2024-09-15T03:17:36.274548+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:36.274548+0000","last_undegraded":"2024-09-15T03:17:36.274548+0000","last_fullsized":"2024-09-15T03:17:36.274548+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":555833,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12049","reported_seq":16225,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:36.314192+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:36.314192+0000","last_peered":"2024-09-15T03:17:36.314192+0000","last_clean":"2024-09-15T03:17:36.314192+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:36.314192+0000","last_undegraded":"2024-09-15T03:17:36.314192+0000","last_fullsized":"2024-09-15T03:17:36.314192+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":686905,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12048,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11621","reported_seq":11723,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.085713+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:37.085713+0000","last_peered":"2024-09-15T03:17:37.085713+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:37.085713+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497266,"num_objects":61,"num_object_clones":0,"num_object_copies":244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11619,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1952}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11830","reported_seq":11938,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.110922+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:37.110922+0000","last_peered":"2024-09-15T03:17:37.110922+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:37.110922+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431730,"num_objects":53,"num_object_clones":0,"num_object_copies":212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11828,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1924}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11951","reported_seq":12063,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:37.191770+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:37.191770+0000","last_peered":"2024-09-15T03:17:37.191770+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:37.191770+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472690,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":8092244,"num_objects":993,"num_object_clones":0,"num_object_copies":3972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189714,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7857,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16211968,"data_stored":16211968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160641,"ondisk_log_size":160641,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927676,"kb_used_data":4276,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93444164,"statfs":{"total":96636764160,"available":95686823936,"internally_reserved":0,"allocated":4378624,"data_stored":4194803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,1,3,1,2,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.441}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.57799999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.69499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":931720,"kb_used_data":4608,"kb_used_omap":1,"kb_used_meta":927102,"kb_avail":93440120,"statfs":{"total":96636764160,"available":95682682880,"internally_reserved":0,"allocated":4718592,"data_stored":4537917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":949352792},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,1,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":7.5},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":6.6029999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.3370000000000002},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.4039999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":2.3730000000000002},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":2.319}]}]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":939328,"kb_used_data":5496,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93432512,"statfs":{"total":96636764160,"available":95674892288,"internally_reserved":0,"allocated":5627904,"data_stored":5444083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,1,7,8,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":14.189},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":11.987}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.181},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.22700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":4.1130000000000004},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.255}]}]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919252,"kb_used_data":4148,"kb_used_omap":4,"kb_used_meta":915067,"kb_avail":93452588,"statfs":{"total":96636764160,"available":95695450112,"internally_reserved":0,"allocated":4247552,"data_stored":4063731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":937029185},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,9,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.27500000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.26200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.22700000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.46800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.436},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.52400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3858432,"data_stored":3858432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4677632,"data_stored":4677632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3870720,"data_stored":3870720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3805184,"data_stored":3805184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:45.981 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:46.281 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:46.283 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:46.295 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-15T03:17:45.516502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645716,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190623,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7916,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160850,"ondisk_log_size":160850,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3686440,"kb_used_data":3656,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373800920,"statfs":{"total":386547056640,"available":382772142080,"internally_reserved":0,"allocated":3743744,"data_stored":3011606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,13,14,7,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14598144,"num_objects":-1782,"num_object_clones":0,"num_object_copies":-7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1782,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.529685"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.346243+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:42.346243+0000","last_peered":"2024-09-15T03:17:42.346243+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:42.346243+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11420","reported_seq":14877,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.127441+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:42.127441+0000","last_peered":"2024-09-15T03:17:42.127441+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:42.127441+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46706,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11418,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12218","reported_seq":12328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.198267+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:42.198267+0000","last_peered":"2024-09-15T03:17:42.198267+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:42.198267+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15185,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12217,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2088}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11617","reported_seq":15591,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:41.630627+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:41.630627+0000","last_peered":"2024-09-15T03:17:41.630627+0000","last_clean":"2024-09-15T03:17:41.630627+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:41.630627+0000","last_undegraded":"2024-09-15T03:17:41.630627+0000","last_fullsized":"2024-09-15T03:17:41.630627+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87666,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12077","reported_seq":12185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.268322+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:42.268322+0000","last_peered":"2024-09-15T03:17:42.268322+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:42.268322+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12075,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11996","reported_seq":12082,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.187758+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:42.187758+0000","last_peered":"2024-09-15T03:17:42.187758+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:42.187758+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11996,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11951","reported_seq":12074,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.105900+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:42.105900+0000","last_peered":"2024-09-15T03:17:42.105900+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:42.105900+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54898,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2019}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12079","reported_seq":12198,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.935603+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:41.935603+0000","last_peered":"2024-09-15T03:17:41.935603+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:41.935603+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46706,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11963","reported_seq":12056,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.033141+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:42.033141+0000","last_peered":"2024-09-15T03:17:42.033141+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:42.033141+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52452,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11960,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.945941+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:41.945941+0000","last_peered":"2024-09-15T03:17:41.945941+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:41.945941+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11962,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11877","reported_seq":11971,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.052739+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:42.052739+0000","last_peered":"2024-09-15T03:17:42.052739+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:42.052739+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45483,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11874,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11889","reported_seq":15932,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:41.514753+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:41.514753+0000","last_peered":"2024-09-15T03:17:41.514753+0000","last_clean":"2024-09-15T03:17:41.514753+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:41.514753+0000","last_undegraded":"2024-09-15T03:17:41.514753+0000","last_fullsized":"2024-09-15T03:17:41.514753+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80697,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12126","reported_seq":16302,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:41.399534+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:41.399534+0000","last_peered":"2024-09-15T03:17:41.399534+0000","last_clean":"2024-09-15T03:17:41.399534+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:41.399534+0000","last_undegraded":"2024-09-15T03:17:41.399534+0000","last_fullsized":"2024-09-15T03:17:41.399534+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56121,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12125,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11782,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.268361+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:42.268361+0000","last_peered":"2024-09-15T03:17:42.268361+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:42.268361+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11879","reported_seq":11988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.998682+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:41.998682+0000","last_peered":"2024-09-15T03:17:41.998682+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:41.998682+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12004","reported_seq":12119,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.130463+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:42.130463+0000","last_peered":"2024-09-15T03:17:42.130463+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:42.130463+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38514,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12002,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":645716,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190623,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7916,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1458176,"data_stored":1458176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160850,"ondisk_log_size":160850,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928280,"kb_used_data":720,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443560,"statfs":{"total":96636764160,"available":95686205440,"internally_reserved":0,"allocated":737280,"data_stored":553459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,5,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.499},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":2.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":22.928999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":10.6},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.36299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915724,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456116,"statfs":{"total":96636764160,"available":95699062784,"internally_reserved":0,"allocated":1118208,"data_stored":937533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.192},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.133},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.1680000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":2.2029999999999998},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":2.2170000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922520,"kb_used_data":1168,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449320,"statfs":{"total":96636764160,"available":95692103680,"internally_reserved":0,"allocated":1196032,"data_stored":1012211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,7,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":4.4130000000000003},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.25}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.5090000000000003},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":4.4699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":3.2149999999999999},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.5490000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919916,"kb_used_data":676,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93451924,"statfs":{"total":96636764160,"available":95694770176,"internally_reserved":0,"allocated":692224,"data_stored":508403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.45600000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.40100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":10.390000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.32500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":225280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:46.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:46.638 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:46.640 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:46.655 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-15T03:17:46.522376+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645716,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190623,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7916,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160850,"ondisk_log_size":160850,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3686440,"kb_used_data":3656,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373800920,"statfs":{"total":386547056640,"available":382772142080,"internally_reserved":0,"allocated":3743744,"data_stored":3011606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,13,14,7,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13205504,"num_objects":-1612,"num_object_clones":0,"num_object_copies":-6448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1612,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.535120"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.346243+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:42.346243+0000","last_peered":"2024-09-15T03:17:42.346243+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:42.346243+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11420","reported_seq":14877,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.127441+0000","last_change":"2024-09-15T03:14:59.821083+0000","last_active":"2024-09-15T03:17:42.127441+0000","last_peered":"2024-09-15T03:17:42.127441+0000","last_clean":"2024-09-15T03:05:04.957543+0000","last_became_active":"2024-09-15T03:06:09.201432+0000","last_became_peered":"2024-09-15T03:06:09.201432+0000","last_unstale":"2024-09-15T03:17:42.127441+0000","last_undegraded":"2024-09-15T03:06:08.881338+0000","last_fullsized":"2024-09-15T03:06:08.841301+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46706,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11418,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":-300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"37'12218","reported_seq":12328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.198267+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:42.198267+0000","last_peered":"2024-09-15T03:17:42.198267+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:42.198267+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15185,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12217,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2088}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11617","reported_seq":15591,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:41.630627+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:41.630627+0000","last_peered":"2024-09-15T03:17:41.630627+0000","last_clean":"2024-09-15T03:17:41.630627+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:41.630627+0000","last_undegraded":"2024-09-15T03:17:41.630627+0000","last_fullsized":"2024-09-15T03:17:41.630627+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87666,"num_objects":11,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11615,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12077","reported_seq":12185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.268322+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:42.268322+0000","last_peered":"2024-09-15T03:17:42.268322+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:42.268322+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12075,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11996","reported_seq":12082,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.187758+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:42.187758+0000","last_peered":"2024-09-15T03:17:42.187758+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:42.187758+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11996,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11951","reported_seq":12074,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.105900+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:42.105900+0000","last_peered":"2024-09-15T03:17:42.105900+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:42.105900+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54898,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2019}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12079","reported_seq":12198,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.935603+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:41.935603+0000","last_peered":"2024-09-15T03:17:41.935603+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:41.935603+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46706,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11963","reported_seq":12056,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.033141+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:42.033141+0000","last_peered":"2024-09-15T03:17:42.033141+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:42.033141+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52452,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11960,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.945941+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:41.945941+0000","last_peered":"2024-09-15T03:17:41.945941+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:41.945941+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11962,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11877","reported_seq":11971,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.052739+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:42.052739+0000","last_peered":"2024-09-15T03:17:42.052739+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:42.052739+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45483,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11874,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11889","reported_seq":15932,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:41.514753+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:41.514753+0000","last_peered":"2024-09-15T03:17:41.514753+0000","last_clean":"2024-09-15T03:17:41.514753+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:41.514753+0000","last_undegraded":"2024-09-15T03:17:41.514753+0000","last_fullsized":"2024-09-15T03:17:41.514753+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80697,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12126","reported_seq":16302,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T03:17:41.399534+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:41.399534+0000","last_peered":"2024-09-15T03:17:41.399534+0000","last_clean":"2024-09-15T03:17:41.399534+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:41.399534+0000","last_undegraded":"2024-09-15T03:17:41.399534+0000","last_fullsized":"2024-09-15T03:17:41.399534+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56121,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12125,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11782,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.268361+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:42.268361+0000","last_peered":"2024-09-15T03:17:42.268361+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:42.268361+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11879","reported_seq":11988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.998682+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:41.998682+0000","last_peered":"2024-09-15T03:17:41.998682+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:41.998682+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12004","reported_seq":12119,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.130463+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:42.130463+0000","last_peered":"2024-09-15T03:17:42.130463+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:42.130463+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38514,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12002,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":645716,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190623,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7916,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1458176,"data_stored":1458176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160850,"ondisk_log_size":160850,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928280,"kb_used_data":720,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443560,"statfs":{"total":96636764160,"available":95686205440,"internally_reserved":0,"allocated":737280,"data_stored":553459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,5,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.499},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":2.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":22.928999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":10.6},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.36299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915724,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456116,"statfs":{"total":96636764160,"available":95699062784,"internally_reserved":0,"allocated":1118208,"data_stored":937533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.192},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.133},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.1680000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":2.2029999999999998},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":2.2170000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922520,"kb_used_data":1168,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449320,"statfs":{"total":96636764160,"available":95692103680,"internally_reserved":0,"allocated":1196032,"data_stored":1012211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,7,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":4.4130000000000003},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.25}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":4.5090000000000003},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":4.4699999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":3.2149999999999999},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":4.5490000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919916,"kb_used_data":676,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93451924,"statfs":{"total":96636764160,"available":95694770176,"internally_reserved":0,"allocated":692224,"data_stored":508403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.45600000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.40100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":10.390000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.32500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":225280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":823296,"data_stored":823296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:46.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:17:46.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11978, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.346243+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:42.346243+0000', 'last_peered': '2024-09-15T03:17:42.346243+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:42.346243+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:46.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T03:17:46.656 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "37'11420", 'reported_seq': 14877, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.127441+0000', 'last_change': '2024-09-15T03:14:59.821083+0000', 'last_active': '2024-09-15T03:17:42.127441+0000', 'last_peered': '2024-09-15T03:17:42.127441+0000', 'last_clean': '2024-09-15T03:05:04.957543+0000', 'last_became_active': '2024-09-15T03:06:09.201432+0000', 'last_became_peered': '2024-09-15T03:06:09.201432+0000', 'last_unstale': '2024-09-15T03:17:42.127441+0000', 'last_undegraded': '2024-09-15T03:06:08.881338+0000', 'last_fullsized': '2024-09-15T03:06:08.841301+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46706, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11418, 'num_write_kb': 45700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1696, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': -300}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12218", 'reported_seq': 12328, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.198267+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:42.198267+0000', 'last_peered': '2024-09-15T03:17:42.198267+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:42.198267+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15185, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12217, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2088}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12077", 'reported_seq': 12185, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.268322+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:42.268322+0000', 'last_peered': '2024-09-15T03:17:42.268322+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:42.268322+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30322, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12075, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2080}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11996", 'reported_seq': 12082, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.187758+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:42.187758+0000', 'last_peered': '2024-09-15T03:17:42.187758+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:42.187758+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16384, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11996, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2036}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11951", 'reported_seq': 12074, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.105900+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:42.105900+0000', 'last_peered': '2024-09-15T03:17:42.105900+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:42.105900+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 54898, 'num_objects': 7, 'num_object_clones': 0, 'num_object_copies': 28, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11949, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2019}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:17:46.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12079", 'reported_seq': 12198, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:41.935603+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:41.935603+0000', 'last_peered': '2024-09-15T03:17:41.935603+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:41.935603+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46706, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2050}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:17:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11963", 'reported_seq': 12056, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.033141+0000', 'last_change': '2024-09-15T03:06:09.181489+0000', 'last_active': '2024-09-15T03:17:42.033141+0000', 'last_peered': '2024-09-15T03:17:42.033141+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:17:42.033141+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52452, 'num_objects': 7, 'num_object_clones': 0, 'num_object_copies': 28, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11960, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2020}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:17:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11964", 'reported_seq': 12085, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:41.945941+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:41.945941+0000', 'last_peered': '2024-09-15T03:17:41.945941+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:41.945941+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30322, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11962, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:17:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11877", 'reported_seq': 11971, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.052739+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:42.052739+0000', 'last_peered': '2024-09-15T03:17:42.052739+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:42.052739+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45483, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11874, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2056}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:17:46.658 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11782, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.268361+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:42.268361+0000', 'last_peered': '2024-09-15T03:17:42.268361+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:42.268361+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:46.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:17:46.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11879", 'reported_seq': 11988, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:41.998682+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:41.998682+0000', 'last_peered': '2024-09-15T03:17:41.998682+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:41.998682+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30322, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:46.659 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:17:46.659 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12004", 'reported_seq': 12119, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:42.130463+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:42.130463+0000', 'last_peered': '2024-09-15T03:17:42.130463+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:42.130463+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38514, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 20, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12002, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:46.659 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:17:47.032 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:47.032 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24768/336 objects degraded (7371.429%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84,"data_bytes":645716,"bytes_used":3774914560,"bytes_avail":382772142080,"bytes_total":386547056640,"degraded_objects":24768,"degraded_total":336,"degraded_ratio":73.714285714285708,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":117,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":148},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:17:47.044 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:17:47.045 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:47.351 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:47.355 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:47.368 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-15T03:17:46.989893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416340,"num_objects":56,"num_object_clones":0,"num_object_copies":224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190651,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8222,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3689952,"kb_used_data":3004,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373797408,"statfs":{"total":386547056640,"available":382768545792,"internally_reserved":0,"allocated":3076096,"data_stored":2339754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,7,6,7,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7675904,"num_objects":-937,"num_object_clones":0,"num_object_copies":-3748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":937,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002172"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11978,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.346243+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:42.346243+0000","last_peered":"2024-09-15T03:17:42.346243+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:42.346243+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12218","reported_seq":12328,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.198267+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:42.198267+0000","last_peered":"2024-09-15T03:17:42.198267+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:42.198267+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15185,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12217,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2088}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12077","reported_seq":12185,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.268322+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:42.268322+0000","last_peered":"2024-09-15T03:17:42.268322+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:42.268322+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12075,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11996","reported_seq":12082,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.187758+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:42.187758+0000","last_peered":"2024-09-15T03:17:42.187758+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:42.187758+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11996,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11951","reported_seq":12074,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.105900+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:42.105900+0000","last_peered":"2024-09-15T03:17:42.105900+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:42.105900+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54898,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11949,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2019}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12079","reported_seq":12198,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.935603+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:41.935603+0000","last_peered":"2024-09-15T03:17:41.935603+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:41.935603+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46706,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2050}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11963","reported_seq":12056,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.033141+0000","last_change":"2024-09-15T03:06:09.181489+0000","last_active":"2024-09-15T03:17:42.033141+0000","last_peered":"2024-09-15T03:17:42.033141+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:42.033141+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52452,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11960,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2020}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11964","reported_seq":12085,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.945941+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:41.945941+0000","last_peered":"2024-09-15T03:17:41.945941+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:41.945941+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11962,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2038}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11877","reported_seq":11971,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.052739+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:42.052739+0000","last_peered":"2024-09-15T03:17:42.052739+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:42.052739+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45483,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11874,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11782,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.268361+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:42.268361+0000","last_peered":"2024-09-15T03:17:42.268361+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:42.268361+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11879","reported_seq":11988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:41.998682+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:41.998682+0000","last_peered":"2024-09-15T03:17:41.998682+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:41.998682+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12004","reported_seq":12119,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:42.130463+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:42.130463+0000","last_peered":"2024-09-15T03:17:42.130463+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:42.130463+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38514,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12002,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":416340,"num_objects":56,"num_object_clones":0,"num_object_copies":224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190651,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8222,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":843776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928280,"kb_used_data":720,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443560,"statfs":{"total":96636764160,"available":95686205440,"internally_reserved":0,"allocated":737280,"data_stored":553459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,3,5,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.499},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":2.2250000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":22.928999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":10.6},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.36299999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915724,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456116,"statfs":{"total":96636764160,"available":95699062784,"internally_reserved":0,"allocated":1118208,"data_stored":937533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.192},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.27400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":2.133},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":2.1680000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":2.2029999999999998},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":2.2170000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926032,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445808,"statfs":{"total":96636764160,"available":95688507392,"internally_reserved":0,"allocated":528384,"data_stored":340359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.28899999999999998},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.34999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919916,"kb_used_data":676,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93451924,"statfs":{"total":96636764160,"available":95694770176,"internally_reserved":0,"allocated":692224,"data_stored":508403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.45600000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.40100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":10.390000000000001},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.32500000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.28399999999999997},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":225280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249856,"data_stored":249856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:50.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:50.713 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:50.727 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-15T03:17:48.990429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8440,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688444,"kb_used_data":1476,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798916,"statfs":{"total":386547056640,"available":382770089984,"internally_reserved":0,"allocated":1511424,"data_stored":754020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7897112,"num_objects":-965,"num_object_clones":0,"num_object_copies":-3860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":965,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002198"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":12498,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:47.379101+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:47.379101+0000","last_peered":"2024-09-15T03:17:47.379101+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:47.379101+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8440,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.46999999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.91100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914948,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456892,"statfs":{"total":96636764160,"available":95699857408,"internally_reserved":0,"allocated":319488,"data_stored":134609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926032,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445808,"statfs":{"total":96636764160,"available":95688507392,"internally_reserved":0,"allocated":528384,"data_stored":340359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.28899999999999998},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.34999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.38}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.436},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.47999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.55300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:50.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:51.062 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:51.065 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:51.075 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-15T03:17:50.991521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8440,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688444,"kb_used_data":1476,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798916,"statfs":{"total":386547056640,"available":382770089984,"internally_reserved":0,"allocated":1511424,"data_stored":754020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450584,"num_objects":-56,"num_object_clones":0,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002821"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":12498,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:47.379101+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:47.379101+0000","last_peered":"2024-09-15T03:17:47.379101+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:47.379101+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8440,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.46999999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.91100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914948,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456892,"statfs":{"total":96636764160,"available":95699857408,"internally_reserved":0,"allocated":319488,"data_stored":134609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926032,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445808,"statfs":{"total":96636764160,"available":95688507392,"internally_reserved":0,"allocated":528384,"data_stored":340359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.28899999999999998},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.34999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.38}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.436},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.47999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.55300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11980, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518101+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:45.518101+0000', 'last_peered': '2024-09-15T03:17:45.518101+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:45.518101+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12332, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527262+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:45.527262+0000', 'last_peered': '2024-09-15T03:17:45.527262+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:45.527262+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12189, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518028+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:45.518028+0000', 'last_peered': '2024-09-15T03:17:45.518028+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:45.518028+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12086, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519218+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:45.519218+0000', 'last_peered': '2024-09-15T03:17:45.519218+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:45.519218+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:51.077 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12081, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522488+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:45.522488+0000', 'last_peered': '2024-09-15T03:17:45.522488+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:45.522488+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12083", 'reported_seq': 12204, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522331+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:45.522331+0000', 'last_peered': '2024-09-15T03:17:45.522331+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:45.522331+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12081, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11966", 'reported_seq': 12498, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:47.379101+0000', 'last_change': '2024-09-15T03:17:44.665811+0000', 'last_active': '2024-09-15T03:17:47.379101+0000', 'last_peered': '2024-09-15T03:17:47.379101+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:17:47.379101+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27876, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1812, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 236, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1808}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 12089, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.528812+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:45.528812+0000', 'last_peered': '2024-09-15T03:17:45.528812+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:45.528812+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11978, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518564+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:45.518564+0000', 'last_peered': '2024-09-15T03:17:45.518564+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:45.518564+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:51.078 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:17:51.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11784, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519826+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:45.519826+0000', 'last_peered': '2024-09-15T03:17:45.519826+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:45.519826+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:51.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:17:51.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11992, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527422+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:45.527422+0000', 'last_peered': '2024-09-15T03:17:45.527422+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:45.527422+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:51.079 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:17:51.079 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12124, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.523656+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:45.523656+0000', 'last_peered': '2024-09-15T03:17:45.523656+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:45.523656+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:51.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:17:51.435 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:51.435 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24244/112 objects degraded (21646.429%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":794,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3776966656,"bytes_avail":382770089984,"bytes_total":386547056640,"degraded_objects":24244,"degraded_total":112,"degraded_ratio":216.46428571428572,"recovering_objects_per_sec":65,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":524,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:17:51.449 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:17:51.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:51.734 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:51.736 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:51.746 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-15T03:17:50.991521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8440,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688444,"kb_used_data":1476,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798916,"statfs":{"total":386547056640,"available":382770089984,"internally_reserved":0,"allocated":1511424,"data_stored":754020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450584,"num_objects":-56,"num_object_clones":0,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002821"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":12498,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:47.379101+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:47.379101+0000","last_peered":"2024-09-15T03:17:47.379101+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:47.379101+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1808}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24244,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8440,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.3779999999999999,"15min":1.651},"min":{"1min":0.191,"5min":0.183,"15min":0.16200000000000001},"max":{"1min":5.3650000000000002,"5min":20.984000000000002,"15min":20.984000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.1640000000000001,"5min":2.1360000000000001,"15min":2.097},"min":{"1min":0.20200000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":9.0280000000000005,"5min":9.6370000000000005,"15min":26.143000000000001},"last":0.46999999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:16:48 2024","interfaces":[{"interface":"back","average":{"1min":3.0089999999999999,"5min":3.6000000000000001,"15min":2.4620000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":13.894,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.04,"5min":1.74,"15min":1.6799999999999999},"min":{"1min":0.23999999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":5.2789999999999999,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.91100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914948,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456892,"statfs":{"total":96636764160,"available":95699857408,"internally_reserved":0,"allocated":319488,"data_stored":134609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926032,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445808,"statfs":{"total":96636764160,"available":95688507392,"internally_reserved":0,"allocated":528384,"data_stored":340359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.28899999999999998},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.26800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.34999999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.27800000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.38}]},{"osd":2,"last update":"Sun Sep 15 03:16:46 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":2.3450000000000002,"15min":2.1669999999999998},"min":{"1min":0.25700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":30.562999999999999,"5min":30.562999999999999,"15min":30.802},"last":0.436},{"interface":"front","average":{"1min":1.8160000000000001,"5min":2.2250000000000001,"15min":1.6180000000000001},"min":{"1min":0.27700000000000002,"5min":0.214,"15min":0.21199999999999999},"max":{"1min":8.8360000000000003,"5min":20.387,"15min":20.387},"last":0.47999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.55300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:54.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:55.067 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:55.071 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:55.084 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-09-15T03:17:54.993042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8864,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692616,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794744,"statfs":{"total":386547056640,"available":382765817856,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450584,"num_objects":-56,"num_object_clones":0,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":948,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.476540"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":13346,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:52.375800+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:52.375800+0000","last_peered":"2024-09-15T03:17:52.375800+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:52.375800+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8864,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.64300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930196,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441644,"statfs":{"total":96636764160,"available":95684243456,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.51200000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.44700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.47999999999999998},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.68999999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.81599999999999995},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.65600000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:55.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:55.408 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:55.412 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:55.424 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-09-15T03:17:54.993042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8864,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692616,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794744,"statfs":{"total":386547056640,"available":382765817856,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450584,"num_objects":-56,"num_object_clones":0,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":948,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.476540"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":13346,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:52.375800+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:52.375800+0000","last_peered":"2024-09-15T03:17:52.375800+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:52.375800+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8864,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.64300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930196,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441644,"statfs":{"total":96636764160,"available":95684243456,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.51200000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.44700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.47999999999999998},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.68999999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.81599999999999995},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.65600000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:55.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:17:55.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11980, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518101+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:45.518101+0000', 'last_peered': '2024-09-15T03:17:45.518101+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:45.518101+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:55.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12332, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527262+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:45.527262+0000', 'last_peered': '2024-09-15T03:17:45.527262+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:45.527262+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12189, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518028+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:45.518028+0000', 'last_peered': '2024-09-15T03:17:45.518028+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:45.518028+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12086, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519218+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:45.519218+0000', 'last_peered': '2024-09-15T03:17:45.519218+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:45.519218+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12081, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522488+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:45.522488+0000', 'last_peered': '2024-09-15T03:17:45.522488+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:45.522488+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12083", 'reported_seq': 12204, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522331+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:45.522331+0000', 'last_peered': '2024-09-15T03:17:45.522331+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:45.522331+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12081, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:55.426 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11966", 'reported_seq': 13346, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:52.375800+0000', 'last_change': '2024-09-15T03:17:44.665811+0000', 'last_active': '2024-09-15T03:17:52.375800+0000', 'last_peered': '2024-09-15T03:17:52.375800+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:17:52.375800+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27876, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1388, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 660, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1384}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 12089, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.528812+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:45.528812+0000', 'last_peered': '2024-09-15T03:17:45.528812+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:45.528812+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11978, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518564+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:45.518564+0000', 'last_peered': '2024-09-15T03:17:45.518564+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:45.518564+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11784, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519826+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:45.519826+0000', 'last_peered': '2024-09-15T03:17:45.519826+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:45.519826+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:17:55.427 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11992, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527422+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:45.527422+0000', 'last_peered': '2024-09-15T03:17:45.527422+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:45.527422+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:55.428 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:17:55.428 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12124, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.523656+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:45.523656+0000', 'last_peered': '2024-09-15T03:17:45.523656+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:45.523656+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:55.428 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:17:55.805 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:55.805 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23820/112 objects degraded (21267.857%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":798,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3781238784,"bytes_avail":382765817856,"bytes_total":386547056640,"degraded_objects":23820,"degraded_total":112,"degraded_ratio":212.67857142857142,"recovering_objects_per_sec":100,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":948,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:17:55.818 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:17:55.819 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:56.131 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:56.133 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:56.145 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-09-15T03:17:54.993042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8864,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692616,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794744,"statfs":{"total":386547056640,"available":382765817856,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-450584,"num_objects":-56,"num_object_clones":0,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-56,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":948,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.476540"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":13346,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:52.375800+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:52.375800+0000","last_peered":"2024-09-15T03:17:52.375800+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:52.375800+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23820,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8864,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.64300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5710000000000002,"5min":2.931,"15min":2.121},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":17.878,"5min":19.388000000000002,"15min":31.175000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.4450000000000001,"15min":1.413},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":6.5659999999999998,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:47 2024","interfaces":[{"interface":"back","average":{"1min":2.0840000000000001,"5min":2.3679999999999999,"15min":1.76},"min":{"1min":0.255,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":7.7789999999999999,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.4689999999999999,"5min":2.5339999999999998,"15min":1.8740000000000001},"min":{"1min":0.27400000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":15.095000000000001,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.59299999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930196,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441644,"statfs":{"total":96636764160,"available":95684243456,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.51200000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.66600000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.44700000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64400000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.47999999999999998},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.68999999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.81599999999999995},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.65600000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:16:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.5509999999999999,"15min":1.387},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":15.114000000000001,"5min":20.199000000000002,"15min":20.948},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.6379999999999999,"5min":2.2250000000000001,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":15.077999999999999,"5min":30.013000000000002,"15min":30.013000000000002},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:59.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:59.491 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:59.503 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-15T03:17:58.994789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9328,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696776,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790584,"statfs":{"total":386547056640,"available":382761558016,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-221208,"num_objects":-28,"num_object_clones":0,"num_object_copies":-112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004895"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":14274,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:57.384778+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:57.384778+0000","last_peered":"2024-09-15T03:17:57.384778+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:57.384778+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9328,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.38},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.438}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.47999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.42499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.51700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.77400000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.628}]}]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437484,"statfs":{"total":96636764160,"available":95679983616,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.64100000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.57599999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.70999999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:59.505 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:17:59.802 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:17:59.805 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:17:59.816 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-15T03:17:58.994789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9328,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696776,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790584,"statfs":{"total":386547056640,"available":382761558016,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-221208,"num_objects":-28,"num_object_clones":0,"num_object_copies":-112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004895"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":14274,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:57.384778+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:57.384778+0000","last_peered":"2024-09-15T03:17:57.384778+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:57.384778+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9328,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.38},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.438}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.47999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.42499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.51700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.77400000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.628}]}]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437484,"statfs":{"total":96636764160,"available":95679983616,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.64100000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.57599999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.70999999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:17:59.817 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:17:59.817 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11980, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518101+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:45.518101+0000', 'last_peered': '2024-09-15T03:17:45.518101+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:45.518101+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12332, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527262+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:45.527262+0000', 'last_peered': '2024-09-15T03:17:45.527262+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:45.527262+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12189, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518028+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:45.518028+0000', 'last_peered': '2024-09-15T03:17:45.518028+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:45.518028+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12086, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519218+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:45.519218+0000', 'last_peered': '2024-09-15T03:17:45.519218+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:45.519218+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12081, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522488+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:45.522488+0000', 'last_peered': '2024-09-15T03:17:45.522488+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:45.522488+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:17:59.818 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12083", 'reported_seq': 12204, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522331+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:45.522331+0000', 'last_peered': '2024-09-15T03:17:45.522331+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:45.522331+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12081, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11966", 'reported_seq': 14274, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:57.384778+0000', 'last_change': '2024-09-15T03:17:44.665811+0000', 'last_active': '2024-09-15T03:17:57.384778+0000', 'last_peered': '2024-09-15T03:17:57.384778+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:17:57.384778+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27876, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 924, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1124, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -920}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 12089, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.528812+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:45.528812+0000', 'last_peered': '2024-09-15T03:17:45.528812+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:45.528812+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11978, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518564+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:45.518564+0000', 'last_peered': '2024-09-15T03:17:45.518564+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:45.518564+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11784, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519826+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:45.519826+0000', 'last_peered': '2024-09-15T03:17:45.519826+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:45.519826+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11992, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527422+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:45.527422+0000', 'last_peered': '2024-09-15T03:17:45.527422+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:45.527422+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:17:59.819 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:17:59.820 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12124, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.523656+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:45.523656+0000', 'last_peered': '2024-09-15T03:17:45.523656+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:45.523656+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:17:59.820 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:00.205 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:00.205 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23356/112 objects degraded (20853.571%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":803,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3785498624,"bytes_avail":382761558016,"bytes_total":386547056640,"degraded_objects":23356,"degraded_total":112,"degraded_ratio":208.53571428571428,"recovering_objects_per_sec":92,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1106,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:00.219 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:00.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:00.526 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:00.528 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:00.540 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-15T03:17:58.994789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9328,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696776,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790584,"statfs":{"total":386547056640,"available":382761558016,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-221208,"num_objects":-28,"num_object_clones":0,"num_object_copies":-112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004895"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":14274,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:57.384778+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:17:57.384778+0000","last_peered":"2024-09-15T03:17:57.384778+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:17:57.384778+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9328,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.38},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.438}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.47999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.42499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.51700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.77400000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.628}]}]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934356,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437484,"statfs":{"total":96636764160,"available":95679983616,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.64100000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.57599999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.70999999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.378}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:03.543 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:03.856 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:03.859 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:03.876 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-15T03:18:02.996304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684296,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373803064,"statfs":{"total":386547056640,"available":382774337536,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004782"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":15400,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:02.382542+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:18:02.382542+0000","last_peered":"2024-09-15T03:18:02.382542+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:18:02.382542+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.60599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.72999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.372}]}]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449964,"statfs":{"total":96636764160,"available":95692763136,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.63600000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.60399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.81799999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.73699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.49099999999999999},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.64600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:03.877 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:04.181 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:04.184 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:04.194 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-15T03:18:02.996304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684296,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373803064,"statfs":{"total":386547056640,"available":382774337536,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004782"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":15400,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:02.382542+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:18:02.382542+0000","last_peered":"2024-09-15T03:18:02.382542+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:18:02.382542+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.60599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.72999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.372}]}]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449964,"statfs":{"total":96636764160,"available":95692763136,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.63600000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.60399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.81799999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.73699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.49099999999999999},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.64600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:04.195 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11980, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518101+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:45.518101+0000', 'last_peered': '2024-09-15T03:17:45.518101+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:45.518101+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12332, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527262+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:45.527262+0000', 'last_peered': '2024-09-15T03:17:45.527262+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:45.527262+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12189, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518028+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:45.518028+0000', 'last_peered': '2024-09-15T03:17:45.518028+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:45.518028+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12086, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519218+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:45.519218+0000', 'last_peered': '2024-09-15T03:17:45.519218+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:45.519218+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12081, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522488+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:45.522488+0000', 'last_peered': '2024-09-15T03:17:45.522488+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:45.522488+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:04.196 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12083", 'reported_seq': 12204, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522331+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:45.522331+0000', 'last_peered': '2024-09-15T03:17:45.522331+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:45.522331+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12081, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11966", 'reported_seq': 15400, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:02.382542+0000', 'last_change': '2024-09-15T03:17:44.665811+0000', 'last_active': '2024-09-15T03:18:02.382542+0000', 'last_peered': '2024-09-15T03:18:02.382542+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:18:02.382542+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27876, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 361, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1687, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -357}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 12089, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.528812+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:45.528812+0000', 'last_peered': '2024-09-15T03:17:45.528812+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:45.528812+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11978, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518564+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:45.518564+0000', 'last_peered': '2024-09-15T03:17:45.518564+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:45.518564+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11784, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519826+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:45.519826+0000', 'last_peered': '2024-09-15T03:17:45.519826+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:45.519826+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:04.197 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:04.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11992, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527422+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:45.527422+0000', 'last_peered': '2024-09-15T03:17:45.527422+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:45.527422+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:04.198 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:04.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12124, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.523656+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:45.523656+0000', 'last_peered': '2024-09-15T03:17:45.523656+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:45.523656+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:04.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:04.568 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:04.568 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22793/112 objects degraded (20350.893%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":807,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3772719104,"bytes_avail":382774337536,"bytes_total":386547056640,"degraded_objects":22793,"degraded_total":112,"degraded_ratio":203.50892857142858,"recovering_objects_per_sec":120,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:04.587 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:04.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:04.878 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:04.881 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:04.892 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-15T03:18:02.996304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684296,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373803064,"statfs":{"total":386547056640,"available":382774337536,"internally_reserved":0,"allocated":1527808,"data_stored":761954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1451,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004782"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14900,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.614763+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.614763+0000","last_peered":"2024-09-15T03:17:46.614763+0000","last_clean":"2024-09-15T03:17:46.614763+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.614763+0000","last_undegraded":"2024-09-15T03:17:46.614763+0000","last_fullsized":"2024-09-15T03:17:46.614763+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15602,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.523967+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:45.523967+0000","last_peered":"2024-09-15T03:17:45.523967+0000","last_clean":"2024-09-15T03:17:45.523967+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:45.523967+0000","last_undegraded":"2024-09-15T03:17:45.523967+0000","last_fullsized":"2024-09-15T03:17:45.523967+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":15400,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:02.382542+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:18:02.382542+0000","last_peered":"2024-09-15T03:18:02.382542+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:18:02.382542+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15943,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524144+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:45.524144+0000","last_peered":"2024-09-15T03:17:45.524144+0000","last_clean":"2024-09-15T03:17:45.524144+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:45.524144+0000","last_undegraded":"2024-09-15T03:17:45.524144+0000","last_fullsized":"2024-09-15T03:17:45.524144+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16310,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-15T03:17:45.524093+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:45.524093+0000","last_peered":"2024-09-15T03:17:45.524093+0000","last_clean":"2024-09-15T03:17:45.524093+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:45.524093+0000","last_undegraded":"2024-09-15T03:17:45.524093+0000","last_fullsized":"2024-09-15T03:17:45.524093+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.60599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.72999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.372}]}]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921876,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449964,"statfs":{"total":96636764160,"available":95692763136,"internally_reserved":0,"allocated":536576,"data_stored":344326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.63600000000000001},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.60399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.81799999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.73699999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.49099999999999999},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.64600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":208896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:07.894 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:08.232 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:08.242 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-15T03:18:07.114495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688460,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798900,"statfs":{"total":386547056640,"available":382770073600,"internally_reserved":0,"allocated":1527808,"data_stored":757713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.119706"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14901,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.615967+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.615967+0000","last_peered":"2024-09-15T03:17:46.615967+0000","last_clean":"2024-09-15T03:17:46.615967+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.615967+0000","last_undegraded":"2024-09-15T03:17:46.615967+0000","last_fullsized":"2024-09-15T03:17:46.615967+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15603,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.527954+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:46.527954+0000","last_peered":"2024-09-15T03:17:46.527954+0000","last_clean":"2024-09-15T03:17:46.527954+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:46.527954+0000","last_undegraded":"2024-09-15T03:17:46.527954+0000","last_fullsized":"2024-09-15T03:17:46.527954+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":15400,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:02.382542+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:18:02.382542+0000","last_peered":"2024-09-15T03:18:02.382542+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:18:02.382542+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15944,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528527+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:46.528527+0000","last_peered":"2024-09-15T03:17:46.528527+0000","last_clean":"2024-09-15T03:17:46.528527+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:46.528527+0000","last_undegraded":"2024-09-15T03:17:46.528527+0000","last_fullsized":"2024-09-15T03:17:46.528527+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16311,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528012+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:46.528012+0000","last_peered":"2024-09-15T03:17:46.528012+0000","last_clean":"2024-09-15T03:17:46.528012+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:46.528012+0000","last_undegraded":"2024-09-15T03:17:46.528012+0000","last_fullsized":"2024-09-15T03:17:46.528012+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.60599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.72999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.372}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926040,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445800,"statfs":{"total":96636764160,"available":95688499200,"internally_reserved":0,"allocated":536576,"data_stored":340085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.436}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.496},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.54300000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.49099999999999999},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.64600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:08.243 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:08.567 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:08.570 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:08.583 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-15T03:18:07.114495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688460,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798900,"statfs":{"total":386547056640,"available":382770073600,"internally_reserved":0,"allocated":1527808,"data_stored":757713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.119706"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11980,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518101+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:17:45.518101+0000","last_peered":"2024-09-15T03:17:45.518101+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:17:45.518101+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14901,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.615967+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.615967+0000","last_peered":"2024-09-15T03:17:46.615967+0000","last_clean":"2024-09-15T03:17:46.615967+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.615967+0000","last_undegraded":"2024-09-15T03:17:46.615967+0000","last_fullsized":"2024-09-15T03:17:46.615967+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12332,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527262+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:17:45.527262+0000","last_peered":"2024-09-15T03:17:45.527262+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:17:45.527262+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15603,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.527954+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:46.527954+0000","last_peered":"2024-09-15T03:17:46.527954+0000","last_clean":"2024-09-15T03:17:46.527954+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:46.527954+0000","last_undegraded":"2024-09-15T03:17:46.527954+0000","last_fullsized":"2024-09-15T03:17:46.527954+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12189,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518028+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:17:45.518028+0000","last_peered":"2024-09-15T03:17:45.518028+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:17:45.518028+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12086,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519218+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:17:45.519218+0000","last_peered":"2024-09-15T03:17:45.519218+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:17:45.519218+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12081,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522488+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:17:45.522488+0000","last_peered":"2024-09-15T03:17:45.522488+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:17:45.522488+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12204,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.522331+0000","last_change":"2024-09-15T03:06:09.272006+0000","last_active":"2024-09-15T03:17:45.522331+0000","last_peered":"2024-09-15T03:17:45.522331+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:17:45.522331+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":15400,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:02.382542+0000","last_change":"2024-09-15T03:17:44.665811+0000","last_active":"2024-09-15T03:18:02.382542+0000","last_peered":"2024-09-15T03:18:02.382542+0000","last_clean":"2024-09-15T03:05:05.107962+0000","last_became_active":"2024-09-15T03:06:09.167408+0000","last_became_peered":"2024-09-15T03:06:09.167408+0000","last_unstale":"2024-09-15T03:18:02.382542+0000","last_undegraded":"2024-09-15T03:06:08.908180+0000","last_fullsized":"2024-09-15T03:06:08.835471+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1687,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-357}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12089,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.528812+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:17:45.528812+0000","last_peered":"2024-09-15T03:17:45.528812+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:17:45.528812+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11978,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.518564+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:17:45.518564+0000","last_peered":"2024-09-15T03:17:45.518564+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:17:45.518564+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15944,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528527+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:46.528527+0000","last_peered":"2024-09-15T03:17:46.528527+0000","last_clean":"2024-09-15T03:17:46.528527+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:46.528527+0000","last_undegraded":"2024-09-15T03:17:46.528527+0000","last_fullsized":"2024-09-15T03:17:46.528527+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16311,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528012+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:46.528012+0000","last_peered":"2024-09-15T03:17:46.528012+0000","last_clean":"2024-09-15T03:17:46.528012+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:46.528012+0000","last_undegraded":"2024-09-15T03:17:46.528012+0000","last_fullsized":"2024-09-15T03:17:46.528012+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11784,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.519826+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:17:45.519826+0000","last_peered":"2024-09-15T03:17:45.519826+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:17:45.519826+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11992,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.527422+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:17:45.527422+0000","last_peered":"2024-09-15T03:17:45.527422+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:17:45.527422+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12124,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:17:45.523656+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:17:45.523656+0000","last_peered":"2024-09-15T03:17:45.523656+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:17:45.523656+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9891,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927892,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443948,"statfs":{"total":96636764160,"available":95686602752,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.56899999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.60599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914956,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456884,"statfs":{"total":96636764160,"available":95699849216,"internally_reserved":0,"allocated":327680,"data_stored":138576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.72999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.372}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926040,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445800,"statfs":{"total":96636764160,"available":95688499200,"internally_reserved":0,"allocated":536576,"data_stored":340085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.436}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.496},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.54300000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919572,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452268,"statfs":{"total":96636764160,"available":95695122432,"internally_reserved":0,"allocated":331776,"data_stored":139526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.49099999999999999},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.64600000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.72499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:08.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:08.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11980, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518101+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:17:45.518101+0000', 'last_peered': '2024-09-15T03:17:45.518101+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:17:45.518101+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:08.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:08.584 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12332, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527262+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:17:45.527262+0000', 'last_peered': '2024-09-15T03:17:45.527262+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:17:45.527262+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:08.584 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12189, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518028+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:17:45.518028+0000', 'last_peered': '2024-09-15T03:17:45.518028+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:17:45.518028+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12086, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519218+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:17:45.519218+0000', 'last_peered': '2024-09-15T03:17:45.519218+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:17:45.519218+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12081, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522488+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:17:45.522488+0000', 'last_peered': '2024-09-15T03:17:45.522488+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:17:45.522488+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12083", 'reported_seq': 12204, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.522331+0000', 'last_change': '2024-09-15T03:06:09.272006+0000', 'last_active': '2024-09-15T03:17:45.522331+0000', 'last_peered': '2024-09-15T03:17:45.522331+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:17:45.522331+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2058, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12081, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11966", 'reported_seq': 15400, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:02.382542+0000', 'last_change': '2024-09-15T03:17:44.665811+0000', 'last_active': '2024-09-15T03:18:02.382542+0000', 'last_peered': '2024-09-15T03:18:02.382542+0000', 'last_clean': '2024-09-15T03:05:05.107962+0000', 'last_became_active': '2024-09-15T03:06:09.167408+0000', 'last_became_peered': '2024-09-15T03:06:09.167408+0000', 'last_unstale': '2024-09-15T03:18:02.382542+0000', 'last_undegraded': '2024-09-15T03:06:08.908180+0000', 'last_fullsized': '2024-09-15T03:06:08.835471+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27876, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 361, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 47865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1687, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -357}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:08.585 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 12089, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.528812+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:17:45.528812+0000', 'last_peered': '2024-09-15T03:17:45.528812+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:17:45.528812+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:08.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:08.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11978, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.518564+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:17:45.518564+0000', 'last_peered': '2024-09-15T03:17:45.518564+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:17:45.518564+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:08.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:08.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11784, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.519826+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:17:45.519826+0000', 'last_peered': '2024-09-15T03:17:45.519826+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:17:45.519826+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:08.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:08.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11992, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.527422+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:17:45.527422+0000', 'last_peered': '2024-09-15T03:17:45.527422+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:17:45.527422+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:08.586 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:08.586 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12124, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:17:45.523656+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:17:45.523656+0000', 'last_peered': '2024-09-15T03:17:45.523656+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:17:45.523656+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:08.587 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:08.966 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:08.966 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22793/112 objects degraded (20350.893%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":812,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3776983040,"bytes_avail":382770073600,"bytes_total":386547056640,"degraded_objects":22793,"degraded_total":112,"degraded_ratio":203.50892857142858,"recovering_objects_per_sec":102,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1027,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:08.980 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:08.980 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:09.306 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:09.308 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:09.319 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-09-15T03:18:08.998366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10516,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688476,"kb_used_data":1516,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798884,"statfs":{"total":386547056640,"available":382770057216,"internally_reserved":0,"allocated":1552384,"data_stored":769566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002872"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14901,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.615967+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.615967+0000","last_peered":"2024-09-15T03:17:46.615967+0000","last_clean":"2024-09-15T03:17:46.615967+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.615967+0000","last_undegraded":"2024-09-15T03:17:46.615967+0000","last_fullsized":"2024-09-15T03:17:46.615967+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15603,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.527954+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:46.527954+0000","last_peered":"2024-09-15T03:17:46.527954+0000","last_clean":"2024-09-15T03:17:46.527954+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:46.527954+0000","last_undegraded":"2024-09-15T03:17:46.527954+0000","last_fullsized":"2024-09-15T03:17:46.527954+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12735,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:07.249000+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:07.249000+0000","last_peered":"2024-09-15T03:18:07.249000+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:07.249000+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15944,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528527+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:46.528527+0000","last_peered":"2024-09-15T03:17:46.528527+0000","last_clean":"2024-09-15T03:17:46.528527+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:46.528527+0000","last_undegraded":"2024-09-15T03:17:46.528527+0000","last_fullsized":"2024-09-15T03:17:46.528527+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16311,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528012+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:46.528012+0000","last_peered":"2024-09-15T03:17:46.528012+0000","last_clean":"2024-09-15T03:17:46.528012+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:46.528012+0000","last_undegraded":"2024-09-15T03:17:46.528012+0000","last_fullsized":"2024-09-15T03:17:46.528012+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10516,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927896,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443944,"statfs":{"total":96636764160,"available":95686598656,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.59199999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.47199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914964,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456876,"statfs":{"total":96636764160,"available":95699841024,"internally_reserved":0,"allocated":335872,"data_stored":142527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.70299999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.751}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.80200000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926040,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445800,"statfs":{"total":96636764160,"available":95688499200,"internally_reserved":0,"allocated":536576,"data_stored":340085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.436}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.496},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.54300000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919576,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452264,"statfs":{"total":96636764160,"available":95695118336,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.39500000000000002},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.63}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:12.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:12.646 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:12.649 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:12.661 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-15T03:18:10.998926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10516,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688476,"kb_used_data":1516,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798884,"statfs":{"total":386547056640,"available":382770057216,"internally_reserved":0,"allocated":1552384,"data_stored":769566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002621"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14901,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.615967+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.615967+0000","last_peered":"2024-09-15T03:17:46.615967+0000","last_clean":"2024-09-15T03:17:46.615967+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.615967+0000","last_undegraded":"2024-09-15T03:17:46.615967+0000","last_fullsized":"2024-09-15T03:17:46.615967+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15603,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.527954+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:46.527954+0000","last_peered":"2024-09-15T03:17:46.527954+0000","last_clean":"2024-09-15T03:17:46.527954+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:46.527954+0000","last_undegraded":"2024-09-15T03:17:46.527954+0000","last_fullsized":"2024-09-15T03:17:46.527954+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12735,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:07.249000+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:07.249000+0000","last_peered":"2024-09-15T03:18:07.249000+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:07.249000+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15944,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528527+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:46.528527+0000","last_peered":"2024-09-15T03:17:46.528527+0000","last_clean":"2024-09-15T03:17:46.528527+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:46.528527+0000","last_undegraded":"2024-09-15T03:17:46.528527+0000","last_fullsized":"2024-09-15T03:17:46.528527+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16311,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528012+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:46.528012+0000","last_peered":"2024-09-15T03:17:46.528012+0000","last_clean":"2024-09-15T03:17:46.528012+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:46.528012+0000","last_undegraded":"2024-09-15T03:17:46.528012+0000","last_fullsized":"2024-09-15T03:17:46.528012+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10516,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927896,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443944,"statfs":{"total":96636764160,"available":95686598656,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.59199999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.47199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914964,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456876,"statfs":{"total":96636764160,"available":95699841024,"internally_reserved":0,"allocated":335872,"data_stored":142527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.70299999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.751}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.80200000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926040,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445800,"statfs":{"total":96636764160,"available":95688499200,"internally_reserved":0,"allocated":536576,"data_stored":340085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.436}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.496},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.54300000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919576,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452264,"statfs":{"total":96636764160,"available":95695118336,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.39500000000000002},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.63}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:12.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:12.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:12.984 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:12.996 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-15T03:18:10.998926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10516,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688476,"kb_used_data":1516,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798884,"statfs":{"total":386547056640,"available":382770057216,"internally_reserved":0,"allocated":1552384,"data_stored":769566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002621"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14901,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.615967+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:17:46.615967+0000","last_peered":"2024-09-15T03:17:46.615967+0000","last_clean":"2024-09-15T03:17:46.615967+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:17:46.615967+0000","last_undegraded":"2024-09-15T03:17:46.615967+0000","last_fullsized":"2024-09-15T03:17:46.615967+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15603,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.527954+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:17:46.527954+0000","last_peered":"2024-09-15T03:17:46.527954+0000","last_clean":"2024-09-15T03:17:46.527954+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:17:46.527954+0000","last_undegraded":"2024-09-15T03:17:46.527954+0000","last_fullsized":"2024-09-15T03:17:46.527954+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12735,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:07.249000+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:07.249000+0000","last_peered":"2024-09-15T03:18:07.249000+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:07.249000+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1794,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15944,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528527+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:17:46.528527+0000","last_peered":"2024-09-15T03:17:46.528527+0000","last_clean":"2024-09-15T03:17:46.528527+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:17:46.528527+0000","last_undegraded":"2024-09-15T03:17:46.528527+0000","last_fullsized":"2024-09-15T03:17:46.528527+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16311,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T03:17:46.528012+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:17:46.528012+0000","last_peered":"2024-09-15T03:17:46.528012+0000","last_clean":"2024-09-15T03:17:46.528012+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:17:46.528012+0000","last_undegraded":"2024-09-15T03:17:46.528012+0000","last_fullsized":"2024-09-15T03:17:46.528012+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10516,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927896,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443944,"statfs":{"total":96636764160,"available":95686598656,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.622}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.59199999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.47199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914964,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456876,"statfs":{"total":96636764160,"available":95699841024,"internally_reserved":0,"allocated":335872,"data_stored":142527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.70299999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.751}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.80200000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926040,"kb_used_data":524,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445800,"statfs":{"total":96636764160,"available":95688499200,"internally_reserved":0,"allocated":536576,"data_stored":340085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.436}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.496},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.54300000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919576,"kb_used_data":332,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452264,"statfs":{"total":96636764160,"available":95695118336,"internally_reserved":0,"allocated":339968,"data_stored":143477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.39500000000000002},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.63}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:12.997 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:12.997 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11982, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105513+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:06.105513+0000', 'last_peered': '2024-09-15T03:18:06.105513+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:06.105513+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12334, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.108909+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:06.108909+0000', 'last_peered': '2024-09-15T03:18:06.108909+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:06.108909+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12191, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.104552+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:06.104552+0000', 'last_peered': '2024-09-15T03:18:06.104552+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:06.104552+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12088, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105437+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:06.105437+0000', 'last_peered': '2024-09-15T03:18:06.105437+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:06.105437+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12083, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.116839+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:18:06.116839+0000', 'last_peered': '2024-09-15T03:18:06.116839+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:06.116839+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:12.998 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:18:12.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12083", 'reported_seq': 12735, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:07.249000+0000', 'last_change': '2024-09-15T03:18:05.152080+0000', 'last_active': '2024-09-15T03:18:07.249000+0000', 'last_peered': '2024-09-15T03:18:07.249000+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:18:07.249000+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1794, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12081, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 291, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:12.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:12.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 12091, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.111438+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:18:06.111438+0000', 'last_peered': '2024-09-15T03:18:06.111438+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:18:06.111438+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:12.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:12.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11980, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.106052+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:06.106052+0000', 'last_peered': '2024-09-15T03:18:06.106052+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:06.106052+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:12.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:12.999 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11786, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105132+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:06.105132+0000', 'last_peered': '2024-09-15T03:18:06.105132+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:06.105132+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:12.999 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:13.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11994, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.107780+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:06.107780+0000', 'last_peered': '2024-09-15T03:18:06.107780+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:06.107780+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:13.000 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:13.000 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12126, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.116987+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:06.116987+0000', 'last_peered': '2024-09-15T03:18:06.116987+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:06.116987+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:13.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:13.378 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:13.378 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21329/112 objects degraded (19043.750%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":816,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3781296128,"bytes_avail":382765760512,"bytes_total":386547056640,"degraded_objects":21329,"degraded_total":112,"degraded_ratio":190.4375,"recovering_objects_per_sec":182,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1464,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:13.393 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:13.393 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:13.688 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:13.691 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:13.706 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-09-15T03:18:12.999675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11355,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692672,"kb_used_data":1548,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794688,"statfs":{"total":386547056640,"available":382765760512,"internally_reserved":0,"allocated":1585152,"data_stored":785286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002551"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14902,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111122+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:06.111122+0000","last_peered":"2024-09-15T03:18:06.111122+0000","last_clean":"2024-09-15T03:18:06.111122+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:06.111122+0000","last_undegraded":"2024-09-15T03:18:06.111122+0000","last_fullsized":"2024-09-15T03:18:06.111122+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15604,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110429+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:06.110429+0000","last_peered":"2024-09-15T03:18:06.110429+0000","last_clean":"2024-09-15T03:18:06.110429+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:06.110429+0000","last_undegraded":"2024-09-15T03:18:06.110429+0000","last_fullsized":"2024-09-15T03:18:06.110429+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":14413,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:12.253193+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:12.253193+0000","last_peered":"2024-09-15T03:18:12.253193+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:12.253193+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15945,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111048+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:06.111048+0000","last_peered":"2024-09-15T03:18:06.111048+0000","last_clean":"2024-09-15T03:18:06.111048+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:06.111048+0000","last_undegraded":"2024-09-15T03:18:06.111048+0000","last_fullsized":"2024-09-15T03:18:06.111048+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110435+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:06.110435+0000","last_peered":"2024-09-15T03:18:06.110435+0000","last_clean":"2024-09-15T03:18:06.110435+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:06.110435+0000","last_undegraded":"2024-09-15T03:18:06.110435+0000","last_fullsized":"2024-09-15T03:18:06.110435+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11355,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927908,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443932,"statfs":{"total":96636764160,"available":95686586368,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.51800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.59499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914972,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456868,"statfs":{"total":96636764160,"available":95699832832,"internally_reserved":0,"allocated":344064,"data_stored":146457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.37}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.5},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46600000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930204,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441636,"statfs":{"total":96636764160,"available":95684235264,"internally_reserved":0,"allocated":544768,"data_stored":344015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.52800000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.34499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.47299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.752},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.50700000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919588,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452252,"statfs":{"total":96636764160,"available":95695106048,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.58499999999999996},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45300000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.48199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.629},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:16.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:17.024 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:17.027 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:17.041 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-15T03:18:17.001024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11355,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688512,"kb_used_data":1548,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798848,"statfs":{"total":386547056640,"available":382770020352,"internally_reserved":0,"allocated":1585152,"data_stored":785286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003198"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14902,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111122+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:06.111122+0000","last_peered":"2024-09-15T03:18:06.111122+0000","last_clean":"2024-09-15T03:18:06.111122+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:06.111122+0000","last_undegraded":"2024-09-15T03:18:06.111122+0000","last_fullsized":"2024-09-15T03:18:06.111122+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15604,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110429+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:06.110429+0000","last_peered":"2024-09-15T03:18:06.110429+0000","last_clean":"2024-09-15T03:18:06.110429+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:06.110429+0000","last_undegraded":"2024-09-15T03:18:06.110429+0000","last_fullsized":"2024-09-15T03:18:06.110429+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":14413,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:12.253193+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:12.253193+0000","last_peered":"2024-09-15T03:18:12.253193+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:12.253193+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15945,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111048+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:06.111048+0000","last_peered":"2024-09-15T03:18:06.111048+0000","last_clean":"2024-09-15T03:18:06.111048+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:06.111048+0000","last_undegraded":"2024-09-15T03:18:06.111048+0000","last_fullsized":"2024-09-15T03:18:06.111048+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110435+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:06.110435+0000","last_peered":"2024-09-15T03:18:06.110435+0000","last_clean":"2024-09-15T03:18:06.110435+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:06.110435+0000","last_undegraded":"2024-09-15T03:18:06.110435+0000","last_fullsized":"2024-09-15T03:18:06.110435+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11355,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927908,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443932,"statfs":{"total":96636764160,"available":95686586368,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.51800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.59499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914972,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456868,"statfs":{"total":96636764160,"available":95699832832,"internally_reserved":0,"allocated":344064,"data_stored":146457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.37}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.5},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46600000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926044,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445796,"statfs":{"total":96636764160,"available":95688495104,"internally_reserved":0,"allocated":544768,"data_stored":344015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.314}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919588,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452252,"statfs":{"total":96636764160,"available":95695106048,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.58499999999999996},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45300000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.48199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.629},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:17.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:17.351 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:17.353 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:17.364 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-15T03:18:17.001024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11355,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688512,"kb_used_data":1548,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798848,"statfs":{"total":386547056640,"available":382770020352,"internally_reserved":0,"allocated":1585152,"data_stored":785286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003198"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14902,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111122+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:06.111122+0000","last_peered":"2024-09-15T03:18:06.111122+0000","last_clean":"2024-09-15T03:18:06.111122+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:06.111122+0000","last_undegraded":"2024-09-15T03:18:06.111122+0000","last_fullsized":"2024-09-15T03:18:06.111122+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15604,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110429+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:06.110429+0000","last_peered":"2024-09-15T03:18:06.110429+0000","last_clean":"2024-09-15T03:18:06.110429+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:06.110429+0000","last_undegraded":"2024-09-15T03:18:06.110429+0000","last_fullsized":"2024-09-15T03:18:06.110429+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":14413,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:12.253193+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:12.253193+0000","last_peered":"2024-09-15T03:18:12.253193+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:12.253193+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15945,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111048+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:06.111048+0000","last_peered":"2024-09-15T03:18:06.111048+0000","last_clean":"2024-09-15T03:18:06.111048+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:06.111048+0000","last_undegraded":"2024-09-15T03:18:06.111048+0000","last_fullsized":"2024-09-15T03:18:06.111048+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110435+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:06.110435+0000","last_peered":"2024-09-15T03:18:06.110435+0000","last_clean":"2024-09-15T03:18:06.110435+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:06.110435+0000","last_undegraded":"2024-09-15T03:18:06.110435+0000","last_fullsized":"2024-09-15T03:18:06.110435+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11355,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927908,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443932,"statfs":{"total":96636764160,"available":95686586368,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.51800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.59499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914972,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456868,"statfs":{"total":96636764160,"available":95699832832,"internally_reserved":0,"allocated":344064,"data_stored":146457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.37}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.5},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46600000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926044,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445796,"statfs":{"total":96636764160,"available":95688495104,"internally_reserved":0,"allocated":544768,"data_stored":344015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.314}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919588,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452252,"statfs":{"total":96636764160,"available":95695106048,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.58499999999999996},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45300000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.48199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.629},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:17.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:17.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11982, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105513+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:06.105513+0000', 'last_peered': '2024-09-15T03:18:06.105513+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:06.105513+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12334, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.108909+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:06.108909+0000', 'last_peered': '2024-09-15T03:18:06.108909+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:06.108909+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12191, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.104552+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:06.104552+0000', 'last_peered': '2024-09-15T03:18:06.104552+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:06.104552+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12088, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105437+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:06.105437+0000', 'last_peered': '2024-09-15T03:18:06.105437+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:06.105437+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12083, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.116839+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:18:06.116839+0000', 'last_peered': '2024-09-15T03:18:06.116839+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:06.116839+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:17.366 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12083", 'reported_seq': 14413, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:12.253193+0000', 'last_change': '2024-09-15T03:18:05.152080+0000', 'last_active': '2024-09-15T03:18:12.253193+0000', 'last_peered': '2024-09-15T03:18:12.253193+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:18:12.253193+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 955, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12081, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1130, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -953}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 12091, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.111438+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:18:06.111438+0000', 'last_peered': '2024-09-15T03:18:06.111438+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:18:06.111438+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11980, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.106052+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:06.106052+0000', 'last_peered': '2024-09-15T03:18:06.106052+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:06.106052+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11786, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105132+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:06.105132+0000', 'last_peered': '2024-09-15T03:18:06.105132+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:06.105132+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11994, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.107780+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:06.107780+0000', 'last_peered': '2024-09-15T03:18:06.107780+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:06.107780+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:17.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12126, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.116987+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:06.116987+0000', 'last_peered': '2024-09-15T03:18:06.116987+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:06.116987+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:17.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:17.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:17.736 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21329/112 objects degraded (19043.750%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":820,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3777036288,"bytes_avail":382770020352,"bytes_total":386547056640,"degraded_objects":21329,"degraded_total":112,"degraded_ratio":190.4375,"recovering_objects_per_sec":146,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1464,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:17.750 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:17.751 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:18.033 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:18.036 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:18.048 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-15T03:18:17.001024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11355,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688512,"kb_used_data":1548,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798848,"statfs":{"total":386547056640,"available":382770020352,"internally_reserved":0,"allocated":1585152,"data_stored":785286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1464,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003198"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14902,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111122+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:06.111122+0000","last_peered":"2024-09-15T03:18:06.111122+0000","last_clean":"2024-09-15T03:18:06.111122+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:06.111122+0000","last_undegraded":"2024-09-15T03:18:06.111122+0000","last_fullsized":"2024-09-15T03:18:06.111122+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15604,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110429+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:06.110429+0000","last_peered":"2024-09-15T03:18:06.110429+0000","last_clean":"2024-09-15T03:18:06.110429+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:06.110429+0000","last_undegraded":"2024-09-15T03:18:06.110429+0000","last_fullsized":"2024-09-15T03:18:06.110429+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":14413,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:12.253193+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:12.253193+0000","last_peered":"2024-09-15T03:18:12.253193+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:12.253193+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1130,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-953}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15945,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111048+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:06.111048+0000","last_peered":"2024-09-15T03:18:06.111048+0000","last_clean":"2024-09-15T03:18:06.111048+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:06.111048+0000","last_undegraded":"2024-09-15T03:18:06.111048+0000","last_fullsized":"2024-09-15T03:18:06.111048+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110435+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:06.110435+0000","last_peered":"2024-09-15T03:18:06.110435+0000","last_clean":"2024-09-15T03:18:06.110435+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:06.110435+0000","last_undegraded":"2024-09-15T03:18:06.110435+0000","last_fullsized":"2024-09-15T03:18:06.110435+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11355,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927908,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443932,"statfs":{"total":96636764160,"available":95686586368,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.57299999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.51800000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.59499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914972,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456868,"statfs":{"total":96636764160,"available":95699832832,"internally_reserved":0,"allocated":344064,"data_stored":146457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.37}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.5},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46600000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926044,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445796,"statfs":{"total":96636764160,"available":95688495104,"internally_reserved":0,"allocated":544768,"data_stored":344015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.314}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919588,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452252,"statfs":{"total":96636764160,"available":95695106048,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.58499999999999996},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45300000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.48199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.629},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.67200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:21.050 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:21.362 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:21.364 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:21.374 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-15T03:18:21.002202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12227,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688512,"kb_used_data":1548,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798848,"statfs":{"total":386547056640,"available":382770020352,"internally_reserved":0,"allocated":1585152,"data_stored":785286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002527"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14902,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111122+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:06.111122+0000","last_peered":"2024-09-15T03:18:06.111122+0000","last_clean":"2024-09-15T03:18:06.111122+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:06.111122+0000","last_undegraded":"2024-09-15T03:18:06.111122+0000","last_fullsized":"2024-09-15T03:18:06.111122+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15604,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110429+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:06.110429+0000","last_peered":"2024-09-15T03:18:06.110429+0000","last_clean":"2024-09-15T03:18:06.110429+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:06.110429+0000","last_undegraded":"2024-09-15T03:18:06.110429+0000","last_fullsized":"2024-09-15T03:18:06.110429+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":16157,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:17.254484+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:17.254484+0000","last_peered":"2024-09-15T03:18:17.254484+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:17.254484+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-81}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15945,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111048+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:06.111048+0000","last_peered":"2024-09-15T03:18:06.111048+0000","last_clean":"2024-09-15T03:18:06.111048+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:06.111048+0000","last_undegraded":"2024-09-15T03:18:06.111048+0000","last_fullsized":"2024-09-15T03:18:06.111048+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110435+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:06.110435+0000","last_peered":"2024-09-15T03:18:06.110435+0000","last_clean":"2024-09-15T03:18:06.110435+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:06.110435+0000","last_undegraded":"2024-09-15T03:18:06.110435+0000","last_fullsized":"2024-09-15T03:18:06.110435+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12227,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927908,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443932,"statfs":{"total":96636764160,"available":95686586368,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.76900000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.68899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914972,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456868,"statfs":{"total":96636764160,"available":95699832832,"internally_reserved":0,"allocated":344064,"data_stored":146457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.503},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.42099999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.68700000000000006}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.64700000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926044,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445796,"statfs":{"total":96636764160,"available":95688495104,"internally_reserved":0,"allocated":544768,"data_stored":344015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.314}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919588,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452252,"statfs":{"total":96636764160,"available":95695106048,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.33800000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.72599999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.80100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:21.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:21.673 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:21.676 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:21.687 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-15T03:18:21.002202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12227,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688512,"kb_used_data":1548,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798848,"statfs":{"total":386547056640,"available":382770020352,"internally_reserved":0,"allocated":1585152,"data_stored":785286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002527"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14902,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111122+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:06.111122+0000","last_peered":"2024-09-15T03:18:06.111122+0000","last_clean":"2024-09-15T03:18:06.111122+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:06.111122+0000","last_undegraded":"2024-09-15T03:18:06.111122+0000","last_fullsized":"2024-09-15T03:18:06.111122+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15604,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110429+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:06.110429+0000","last_peered":"2024-09-15T03:18:06.110429+0000","last_clean":"2024-09-15T03:18:06.110429+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:06.110429+0000","last_undegraded":"2024-09-15T03:18:06.110429+0000","last_fullsized":"2024-09-15T03:18:06.110429+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":16157,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:17.254484+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:17.254484+0000","last_peered":"2024-09-15T03:18:17.254484+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:17.254484+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-81}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15945,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111048+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:06.111048+0000","last_peered":"2024-09-15T03:18:06.111048+0000","last_clean":"2024-09-15T03:18:06.111048+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:06.111048+0000","last_undegraded":"2024-09-15T03:18:06.111048+0000","last_fullsized":"2024-09-15T03:18:06.111048+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110435+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:06.110435+0000","last_peered":"2024-09-15T03:18:06.110435+0000","last_clean":"2024-09-15T03:18:06.110435+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:06.110435+0000","last_undegraded":"2024-09-15T03:18:06.110435+0000","last_fullsized":"2024-09-15T03:18:06.110435+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12227,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927908,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443932,"statfs":{"total":96636764160,"available":95686586368,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.76900000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.68899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914972,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456868,"statfs":{"total":96636764160,"available":95699832832,"internally_reserved":0,"allocated":344064,"data_stored":146457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.503},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.42099999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.68700000000000006}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.64700000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926044,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445796,"statfs":{"total":96636764160,"available":95688495104,"internally_reserved":0,"allocated":544768,"data_stored":344015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.314}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919588,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452252,"statfs":{"total":96636764160,"available":95695106048,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.33800000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.72599999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.80100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:21.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:21.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11982, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105513+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:06.105513+0000', 'last_peered': '2024-09-15T03:18:06.105513+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:06.105513+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:21.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:21.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12334, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.108909+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:06.108909+0000', 'last_peered': '2024-09-15T03:18:06.108909+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:06.108909+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:21.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:21.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12191, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.104552+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:06.104552+0000', 'last_peered': '2024-09-15T03:18:06.104552+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:06.104552+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12088, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105437+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:06.105437+0000', 'last_peered': '2024-09-15T03:18:06.105437+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:06.105437+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12083, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.116839+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:18:06.116839+0000', 'last_peered': '2024-09-15T03:18:06.116839+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:06.116839+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12083", 'reported_seq': 16157, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:17.254484+0000', 'last_change': '2024-09-15T03:18:05.152080+0000', 'last_active': '2024-09-15T03:18:17.254484+0000', 'last_peered': '2024-09-15T03:18:17.254484+0000', 'last_clean': '2024-09-15T03:05:05.255845+0000', 'last_became_active': '2024-09-15T03:06:09.253359+0000', 'last_became_peered': '2024-09-15T03:06:09.253359+0000', 'last_unstale': '2024-09-15T03:18:17.254484+0000', 'last_undegraded': '2024-09-15T03:06:08.887546+0000', 'last_fullsized': '2024-09-15T03:06:08.840523+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 83, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12081, 'num_write_kb': 48332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2002, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -81}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 12091, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.111438+0000', 'last_change': '2024-09-15T03:06:09.233052+0000', 'last_active': '2024-09-15T03:18:06.111438+0000', 'last_peered': '2024-09-15T03:18:06.111438+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:18:06.111438+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2044, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2040}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11980, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.106052+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:06.106052+0000', 'last_peered': '2024-09-15T03:18:06.106052+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:06.106052+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:21.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:21.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11786, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.105132+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:06.105132+0000', 'last_peered': '2024-09-15T03:18:06.105132+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:06.105132+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:21.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:21.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11994, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.107780+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:06.107780+0000', 'last_peered': '2024-09-15T03:18:06.107780+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:06.107780+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:21.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:21.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12126, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:06.116987+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:06.116987+0000', 'last_peered': '2024-09-15T03:18:06.116987+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:06.116987+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:21.690 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:22.047 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:22.047 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20457/112 objects degraded (18265.179%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":825,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3777036288,"bytes_avail":382770020352,"bytes_total":386547056640,"degraded_objects":20457,"degraded_total":112,"degraded_ratio":182.65178571428572,"recovering_objects_per_sec":108,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":872,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:22.061 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:22.061 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:22.385 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:22.389 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:22.402 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-15T03:18:21.002202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12227,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688512,"kb_used_data":1548,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798848,"statfs":{"total":386547056640,"available":382770020352,"internally_reserved":0,"allocated":1585152,"data_stored":785286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-872,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002527"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11982,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105513+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:06.105513+0000","last_peered":"2024-09-15T03:18:06.105513+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:06.105513+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14902,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111122+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:06.111122+0000","last_peered":"2024-09-15T03:18:06.111122+0000","last_clean":"2024-09-15T03:18:06.111122+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:06.111122+0000","last_undegraded":"2024-09-15T03:18:06.111122+0000","last_fullsized":"2024-09-15T03:18:06.111122+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12334,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.108909+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:06.108909+0000","last_peered":"2024-09-15T03:18:06.108909+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:06.108909+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15604,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110429+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:06.110429+0000","last_peered":"2024-09-15T03:18:06.110429+0000","last_clean":"2024-09-15T03:18:06.110429+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:06.110429+0000","last_undegraded":"2024-09-15T03:18:06.110429+0000","last_fullsized":"2024-09-15T03:18:06.110429+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12191,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.104552+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:06.104552+0000","last_peered":"2024-09-15T03:18:06.104552+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:06.104552+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12088,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105437+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:06.105437+0000","last_peered":"2024-09-15T03:18:06.105437+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:06.105437+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12083,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116839+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:06.116839+0000","last_peered":"2024-09-15T03:18:06.116839+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:06.116839+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":16157,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:17.254484+0000","last_change":"2024-09-15T03:18:05.152080+0000","last_active":"2024-09-15T03:18:17.254484+0000","last_peered":"2024-09-15T03:18:17.254484+0000","last_clean":"2024-09-15T03:05:05.255845+0000","last_became_active":"2024-09-15T03:06:09.253359+0000","last_became_peered":"2024-09-15T03:06:09.253359+0000","last_unstale":"2024-09-15T03:18:17.254484+0000","last_undegraded":"2024-09-15T03:06:08.887546+0000","last_fullsized":"2024-09-15T03:06:08.840523+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":83,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-81}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16136,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T03:18:07.256839+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:07.256839+0000","last_peered":"2024-09-15T03:18:07.256839+0000","last_clean":"2024-09-15T03:18:07.256839+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:07.256839+0000","last_undegraded":"2024-09-15T03:18:07.256839+0000","last_fullsized":"2024-09-15T03:18:07.256839+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":12091,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.111438+0000","last_change":"2024-09-15T03:06:09.233052+0000","last_active":"2024-09-15T03:18:06.111438+0000","last_peered":"2024-09-15T03:18:06.111438+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:06.111438+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2040}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11980,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.106052+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:06.106052+0000","last_peered":"2024-09-15T03:18:06.106052+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:06.106052+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15945,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.111048+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:06.111048+0000","last_peered":"2024-09-15T03:18:06.111048+0000","last_clean":"2024-09-15T03:18:06.111048+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:06.111048+0000","last_undegraded":"2024-09-15T03:18:06.111048+0000","last_fullsized":"2024-09-15T03:18:06.111048+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16312,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T03:18:06.110435+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:06.110435+0000","last_peered":"2024-09-15T03:18:06.110435+0000","last_clean":"2024-09-15T03:18:06.110435+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:06.110435+0000","last_undegraded":"2024-09-15T03:18:06.110435+0000","last_fullsized":"2024-09-15T03:18:06.110435+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11786,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.105132+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:06.105132+0000","last_peered":"2024-09-15T03:18:06.105132+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:06.105132+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11994,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.107780+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:06.107780+0000","last_peered":"2024-09-15T03:18:06.107780+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:06.107780+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12126,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:06.116987+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:06.116987+0000","last_peered":"2024-09-15T03:18:06.116987+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:06.116987+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12227,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":212992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927908,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443932,"statfs":{"total":96636764160,"available":95686586368,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.76900000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.68899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914972,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456868,"statfs":{"total":96636764160,"available":95699832832,"internally_reserved":0,"allocated":344064,"data_stored":146457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.503},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.42099999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.68700000000000006}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.64700000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926044,"kb_used_data":532,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445796,"statfs":{"total":96636764160,"available":95688495104,"internally_reserved":0,"allocated":544768,"data_stored":344015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":4.9530000000000003,"5min":4.1360000000000001,"15min":2.5310000000000001},"min":{"1min":0.23499999999999999,"5min":0.17399999999999999,"15min":0.16900000000000001},"max":{"1min":28.456,"5min":28.456,"15min":28.456},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.427,"5min":2.4180000000000001,"15min":2.02},"min":{"1min":0.19,"5min":0.19,"15min":0.187},"max":{"1min":12.271000000000001,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.9420000000000002,"15min":2.0920000000000001},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":4.21,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":3.0139999999999998,"5min":4.2960000000000003,"15min":2.8180000000000001},"min":{"1min":0.217,"5min":0.20899999999999999,"15min":0.20000000000000001},"max":{"1min":17.332000000000001,"5min":28.145,"15min":28.145},"last":0.314}]},{"osd":3,"last update":"Sun Sep 15 03:17:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8660000000000001,"5min":4.4400000000000004,"15min":2.657},"min":{"1min":0.255,"5min":0.249,"15min":0.222},"max":{"1min":34.595999999999997,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.8660000000000001,"5min":2.3700000000000001,"15min":1.8200000000000001},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22700000000000001},"max":{"1min":19.356000000000002,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40899999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919588,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452252,"statfs":{"total":96636764160,"available":95695106048,"internally_reserved":0,"allocated":348160,"data_stored":147407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.33800000000000002},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.72599999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.80100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":200704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:25.404 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:25.727 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:25.729 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:25.741 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-15T03:18:25.003539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12868,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692728,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794632,"statfs":{"total":386547056640,"available":382765703168,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002514"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":13210,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:22.211442+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:22.211442+0000","last_peered":"2024-09-15T03:18:22.211442+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:22.211442+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1484}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12868,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.373},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.316}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.46400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.28899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.318},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.39100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.55500000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930212,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441628,"statfs":{"total":96636764160,"available":95684227072,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.47399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.52200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.501}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.40500000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.41499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45200000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.38},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:25.742 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:26.037 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:26.039 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:26.051 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-15T03:18:25.003539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12868,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692728,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794632,"statfs":{"total":386547056640,"available":382765703168,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002514"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":13210,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:22.211442+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:22.211442+0000","last_peered":"2024-09-15T03:18:22.211442+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:22.211442+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1484}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12868,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.373},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.316}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.46400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.28899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.318},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.39100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.55500000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930212,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441628,"statfs":{"total":96636764160,"available":95684227072,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.47399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.52200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.501}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.40500000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.41499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45200000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.38},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11984, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.168297+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:18.168297+0000', 'last_peered': '2024-09-15T03:18:18.168297+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:18.168297+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12336, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.169386+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:18.169386+0000', 'last_peered': '2024-09-15T03:18:18.169386+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:18.169386+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12193, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.164830+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:18.164830+0000', 'last_peered': '2024-09-15T03:18:18.164830+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:18.164830+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12090, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166563+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:18.166563+0000', 'last_peered': '2024-09-15T03:18:18.166563+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:18.166563+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:26.053 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12085, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.171878+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:18:18.171878+0000', 'last_peered': '2024-09-15T03:18:18.171878+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:18.171878+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 13210, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:22.211442+0000', 'last_change': '2024-09-15T03:18:17.693323+0000', 'last_active': '2024-09-15T03:18:22.211442+0000', 'last_peered': '2024-09-15T03:18:22.211442+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:18:22.211442+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1486, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 576, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1484}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11982, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166727+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:18.166727+0000', 'last_peered': '2024-09-15T03:18:18.166727+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:18.166727+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11788, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166690+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:18.166690+0000', 'last_peered': '2024-09-15T03:18:18.166690+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:18.166690+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11996, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.165623+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:18.165623+0000', 'last_peered': '2024-09-15T03:18:18.165623+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:18.165623+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:26.054 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12128, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.177513+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:18.177513+0000', 'last_peered': '2024-09-15T03:18:18.177513+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:18.177513+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:26.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:26.419 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:26.419 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19816/112 objects degraded (17692.857%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":829,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3781353472,"bytes_avail":382765703168,"bytes_total":386547056640,"degraded_objects":19816,"degraded_total":112,"degraded_ratio":176.92857142857142,"recovering_objects_per_sec":189,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1513,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:26.434 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:26.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:26.714 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:26.727 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-15T03:18:25.003539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12868,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692728,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794632,"statfs":{"total":386547056640,"available":382765703168,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1513,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002514"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":13210,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:22.211442+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:22.211442+0000","last_peered":"2024-09-15T03:18:22.211442+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:22.211442+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1484}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12868,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.373},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.316}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.46400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.28899999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:17:22 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.7400000000000002,"15min":1.804},"min":{"1min":0.30099999999999999,"5min":0.219,"15min":0.19900000000000001},"max":{"1min":5.851,"5min":32.295999999999999,"15min":32.295999999999999},"last":0.318},{"interface":"front","average":{"1min":1.53,"5min":2.7839999999999998,"15min":1.861},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.20999999999999999},"max":{"1min":5.8879999999999999,"5min":32.340000000000003,"15min":32.340000000000003},"last":0.39100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.55500000000000005}]}]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930212,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441628,"statfs":{"total":96636764160,"available":95684227072,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.47399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.52200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.501}]}]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.40500000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.41499999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45200000000000001},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.29099999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.38},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.435}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:29.729 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:30.053 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:30.055 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:30.066 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-15T03:18:29.005184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13327,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696888,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790472,"statfs":{"total":386547056640,"available":382761443328,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003674"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":14128,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:27.210671+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:27.210671+0000","last_peered":"2024-09-15T03:18:27.210671+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:27.210671+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1035,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1025}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13327,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.84199999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":2.4119999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.68400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.628},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.45300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934372,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437468,"statfs":{"total":96636764160,"available":95679967232,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.77700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.62}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.71299999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.76300000000000001},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.81599999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.44400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:30.068 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:30.366 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:30.369 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:30.380 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-15T03:18:29.005184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13327,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696888,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790472,"statfs":{"total":386547056640,"available":382761443328,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003674"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":14128,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:27.210671+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:27.210671+0000","last_peered":"2024-09-15T03:18:27.210671+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:27.210671+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1035,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1025}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13327,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.84199999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":2.4119999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.68400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.628},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.45300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934372,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437468,"statfs":{"total":96636764160,"available":95679967232,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.77700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.62}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.71299999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.76300000000000001},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.81599999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.44400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:30.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11984, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.168297+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:18.168297+0000', 'last_peered': '2024-09-15T03:18:18.168297+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:18.168297+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12336, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.169386+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:18.169386+0000', 'last_peered': '2024-09-15T03:18:18.169386+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:18.169386+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12193, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.164830+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:18.164830+0000', 'last_peered': '2024-09-15T03:18:18.164830+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:18.164830+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12090, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166563+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:18.166563+0000', 'last_peered': '2024-09-15T03:18:18.166563+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:18.166563+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:30.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12085, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.171878+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:18:18.171878+0000', 'last_peered': '2024-09-15T03:18:18.171878+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:18.171878+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 14128, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:27.210671+0000', 'last_change': '2024-09-15T03:18:17.693323+0000', 'last_active': '2024-09-15T03:18:27.210671+0000', 'last_peered': '2024-09-15T03:18:27.210671+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:18:27.210671+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1027, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1035, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1025}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11982, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166727+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:18.166727+0000', 'last_peered': '2024-09-15T03:18:18.166727+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:18.166727+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11788, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166690+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:18.166690+0000', 'last_peered': '2024-09-15T03:18:18.166690+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:18.166690+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11996, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.165623+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:18.165623+0000', 'last_peered': '2024-09-15T03:18:18.165623+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:18.165623+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:30.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12128, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.177513+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:18.177513+0000', 'last_peered': '2024-09-15T03:18:18.177513+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:18.177513+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:30.384 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:30.773 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:30.774 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19357/112 objects degraded (17283.036%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":833,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3785613312,"bytes_avail":382761443328,"bytes_total":386547056640,"degraded_objects":19357,"degraded_total":112,"degraded_ratio":172.83035714285714,"recovering_objects_per_sec":109,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1100,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:30.788 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:30.788 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:31.063 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:31.066 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:31.076 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-15T03:18:31.005694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13327,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696888,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790472,"statfs":{"total":386547056640,"available":382761443328,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1100,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.834492"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":14128,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:27.210671+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:27.210671+0000","last_peered":"2024-09-15T03:18:27.210671+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:27.210671+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1035,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1025}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13327,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.84199999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":2.4119999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.52200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.68400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.628},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.45300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934372,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437468,"statfs":{"total":96636764160,"available":95679967232,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.77700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.62}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.71299999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.76300000000000001},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.81599999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.44400000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:34.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:34.395 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:34.398 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:34.410 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-15T03:18:33.006349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13848,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684408,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373802952,"statfs":{"total":386547056640,"available":382774222848,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1621,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004146"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":15170,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:32.211908+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:32.211908+0000","last_peered":"2024-09-15T03:18:32.211908+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:32.211908+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-504}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13848,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.84199999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":2.4119999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.35699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.57299999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.42199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921892,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449948,"statfs":{"total":96636764160,"available":95692746752,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.44}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.621}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.40500000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.60399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.53800000000000003},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:34.414 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:34.701 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:34.703 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:34.713 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-15T03:18:33.006349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13848,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684408,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373802952,"statfs":{"total":386547056640,"available":382774222848,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1621,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004146"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":15170,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:32.211908+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:32.211908+0000","last_peered":"2024-09-15T03:18:32.211908+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:32.211908+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-504}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13848,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.84199999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":2.4119999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.35699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.57299999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.42199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921892,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449948,"statfs":{"total":96636764160,"available":95692746752,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.44}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.621}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.40500000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.60399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.53800000000000003},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:34.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:34.715 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11984, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.168297+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:18.168297+0000', 'last_peered': '2024-09-15T03:18:18.168297+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:18.168297+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:34.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:34.715 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12336, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.169386+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:18.169386+0000', 'last_peered': '2024-09-15T03:18:18.169386+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:18.169386+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:34.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:34.715 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12193, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.164830+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:18.164830+0000', 'last_peered': '2024-09-15T03:18:18.164830+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:18.164830+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:34.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:34.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12090, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166563+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:18.166563+0000', 'last_peered': '2024-09-15T03:18:18.166563+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:18.166563+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:34.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:34.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12085, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.171878+0000', 'last_change': '2024-09-15T03:06:09.313727+0000', 'last_active': '2024-09-15T03:18:18.171878+0000', 'last_peered': '2024-09-15T03:18:18.171878+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:18.171878+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2028, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:34.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:34.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 15170, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:32.211908+0000', 'last_change': '2024-09-15T03:18:17.693323+0000', 'last_active': '2024-09-15T03:18:32.211908+0000', 'last_peered': '2024-09-15T03:18:32.211908+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:18:32.211908+0000', 'last_undegraded': '2024-09-15T03:06:08.924294+0000', 'last_fullsized': '2024-09-15T03:06:08.845678+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 506, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1556, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -504}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:34.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:34.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11982, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166727+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:18.166727+0000', 'last_peered': '2024-09-15T03:18:18.166727+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:18.166727+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:34.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:34.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11788, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.166690+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:18.166690+0000', 'last_peered': '2024-09-15T03:18:18.166690+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:18.166690+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:34.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:34.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11996, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.165623+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:18.165623+0000', 'last_peered': '2024-09-15T03:18:18.165623+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:18.165623+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:34.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:34.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12128, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:18.177513+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:18.177513+0000', 'last_peered': '2024-09-15T03:18:18.177513+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:18.177513+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:34.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:35.100 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:35.100 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18836/112 objects degraded (16817.857%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":838,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3772833792,"bytes_avail":382774222848,"bytes_total":386547056640,"degraded_objects":18836,"degraded_total":112,"degraded_ratio":168.17857142857142,"recovering_objects_per_sec":135,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1621,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:35.119 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:35.119 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:35.416 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:35.418 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:35.428 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-09-15T03:18:35.007058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13848,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3684408,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":3682737,"kb_avail":373802952,"statfs":{"total":386547056640,"available":382774222848,"internally_reserved":0,"allocated":1642496,"data_stored":808322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3771122715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004184"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14904,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169227+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:18.169227+0000","last_peered":"2024-09-15T03:18:18.169227+0000","last_clean":"2024-09-15T03:18:18.169227+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:18.169227+0000","last_undegraded":"2024-09-15T03:18:18.169227+0000","last_fullsized":"2024-09-15T03:18:18.169227+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12336,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.169386+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:18.169386+0000","last_peered":"2024-09-15T03:18:18.169386+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:18.169386+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15606,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168011+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:18.168011+0000","last_peered":"2024-09-15T03:18:18.168011+0000","last_clean":"2024-09-15T03:18:18.168011+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:18.168011+0000","last_undegraded":"2024-09-15T03:18:18.168011+0000","last_fullsized":"2024-09-15T03:18:18.168011+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.268917+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.268917+0000","last_peered":"2024-09-15T03:18:19.268917+0000","last_clean":"2024-09-15T03:18:19.268917+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.268917+0000","last_undegraded":"2024-09-15T03:18:19.268917+0000","last_fullsized":"2024-09-15T03:18:19.268917+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":15170,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:32.211908+0000","last_change":"2024-09-15T03:18:17.693323+0000","last_active":"2024-09-15T03:18:32.211908+0000","last_peered":"2024-09-15T03:18:32.211908+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:32.211908+0000","last_undegraded":"2024-09-15T03:06:08.924294+0000","last_fullsized":"2024-09-15T03:06:08.845678+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-504}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.169049+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:18.169049+0000","last_peered":"2024-09-15T03:18:18.169049+0000","last_clean":"2024-09-15T03:18:18.169049+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:18.169049+0000","last_undegraded":"2024-09-15T03:18:18.169049+0000","last_fullsized":"2024-09-15T03:18:18.169049+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16314,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.168197+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:18.168197+0000","last_peered":"2024-09-15T03:18:18.168197+0000","last_clean":"2024-09-15T03:18:18.168197+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:18.168197+0000","last_undegraded":"2024-09-15T03:18:18.168197+0000","last_fullsized":"2024-09-15T03:18:18.168197+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11996,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.165623+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:18.165623+0000","last_peered":"2024-09-15T03:18:18.165623+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:18.165623+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18836,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13848,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":204800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.84199999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":2.4119999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914988,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456852,"statfs":{"total":96636764160,"available":95699816448,"internally_reserved":0,"allocated":360448,"data_stored":154264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.35699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.57299999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.42199999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921892,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449948,"statfs":{"total":96636764160,"available":95692746752,"internally_reserved":0,"allocated":552960,"data_stored":343630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.44}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.621}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.40500000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.60399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.53800000000000003},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":192512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:38.430 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:38.765 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:38.767 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:38.778 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-15T03:18:37.238526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14354,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688580,"kb_used_data":1612,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798780,"statfs":{"total":386547056640,"available":382769950720,"internally_reserved":0,"allocated":1650688,"data_stored":807884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.233341"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11984,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.168297+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:18.168297+0000","last_peered":"2024-09-15T03:18:18.168297+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:18.168297+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14905,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.174349+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:19.174349+0000","last_peered":"2024-09-15T03:18:19.174349+0000","last_clean":"2024-09-15T03:18:19.174349+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:19.174349+0000","last_undegraded":"2024-09-15T03:18:19.174349+0000","last_fullsized":"2024-09-15T03:18:19.174349+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12337,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:19.177518+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:19.177518+0000","last_peered":"2024-09-15T03:18:19.177518+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:19.177518+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15607,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.173211+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:19.173211+0000","last_peered":"2024-09-15T03:18:19.173211+0000","last_clean":"2024-09-15T03:18:19.173211+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:19.173211+0000","last_undegraded":"2024-09-15T03:18:19.173211+0000","last_fullsized":"2024-09-15T03:18:19.173211+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12193,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.164830+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:18.164830+0000","last_peered":"2024-09-15T03:18:18.164830+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:18.164830+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12090,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166563+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:18.166563+0000","last_peered":"2024-09-15T03:18:18.166563+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:18.166563+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12085,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.171878+0000","last_change":"2024-09-15T03:06:09.313727+0000","last_active":"2024-09-15T03:18:18.171878+0000","last_peered":"2024-09-15T03:18:18.171878+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:18.171878+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12218,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.269417+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.269417+0000","last_peered":"2024-09-15T03:18:19.269417+0000","last_clean":"2024-09-15T03:18:19.269417+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.269417+0000","last_undegraded":"2024-09-15T03:18:19.269417+0000","last_fullsized":"2024-09-15T03:18:19.269417+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16138,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-15T03:18:18.164838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:18.164838+0000","last_peered":"2024-09-15T03:18:18.164838+0000","last_clean":"2024-09-15T03:18:18.164838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:18.164838+0000","last_undegraded":"2024-09-15T03:18:18.164838+0000","last_fullsized":"2024-09-15T03:18:18.164838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16186,"reported_epoch":44,"state":"peering","last_fresh":"2024-09-15T03:18:36.237236+0000","last_change":"2024-09-15T03:18:36.240398+0000","last_active":"2024-09-15T03:18:36.237209+0000","last_peered":"2024-09-15T03:18:36.147637+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:36.237236+0000","last_undegraded":"2024-09-15T03:18:36.237236+0000","last_fullsized":"2024-09-15T03:18:36.237236+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11982,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166727+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:18.166727+0000","last_peered":"2024-09-15T03:18:18.166727+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:18.166727+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15948,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.174512+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:19.174512+0000","last_peered":"2024-09-15T03:18:19.174512+0000","last_clean":"2024-09-15T03:18:19.174512+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:19.174512+0000","last_undegraded":"2024-09-15T03:18:19.174512+0000","last_fullsized":"2024-09-15T03:18:19.174512+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16315,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.173417+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:19.173417+0000","last_peered":"2024-09-15T03:18:19.173417+0000","last_clean":"2024-09-15T03:18:19.173417+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:19.173417+0000","last_undegraded":"2024-09-15T03:18:19.173417+0000","last_fullsized":"2024-09-15T03:18:19.173417+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11788,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.166690+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:18.166690+0000","last_peered":"2024-09-15T03:18:18.166690+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:18.166690+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11997,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:19.177309+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:19.177309+0000","last_peered":"2024-09-15T03:18:19.177309+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:19.177309+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:18.177513+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:18.177513+0000","last_peered":"2024-09-15T03:18:18.177513+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:18.177513+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14354,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927924,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443916,"statfs":{"total":96636764160,"available":95686569984,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.84199999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:17:34 2024","interfaces":[{"interface":"back","average":{"1min":4.9690000000000003,"5min":4.5199999999999996,"15min":2.927},"min":{"1min":0.249,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":26.373000000000001,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":3.7290000000000001,"5min":3.1640000000000001,"15min":2.016},"min":{"1min":0.27700000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":20.097999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":2.4119999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.63600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914996,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456844,"statfs":{"total":96636764160,"available":95699808256,"internally_reserved":0,"allocated":368640,"data_stored":158141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926056,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445784,"statfs":{"total":96636764160,"available":95688482816,"internally_reserved":0,"allocated":552960,"data_stored":339315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.439}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.63},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.753}]}]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919604,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452236,"statfs":{"total":96636764160,"available":95695089664,"internally_reserved":0,"allocated":364544,"data_stored":155214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:17:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":2.2909999999999999,"15min":1.6599999999999999},"min":{"1min":0.26700000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":20.643999999999998,"5min":22.384,"15min":22.384},"last":0.40500000000000003},{"interface":"front","average":{"1min":3.5579999999999998,"5min":3.3420000000000001,"15min":2.657},"min":{"1min":0.27700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":20.353000000000002,"5min":30.433,"15min":30.433},"last":0.60399999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.53800000000000003},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.55100000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.76100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:38.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:39.100 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:39.103 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:39.114 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-15T03:18:39.008420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14439,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688588,"kb_used_data":1628,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798772,"statfs":{"total":386547056640,"available":382769942528,"internally_reserved":0,"allocated":1667072,"data_stored":815638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002725"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14905,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.174349+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:19.174349+0000","last_peered":"2024-09-15T03:18:19.174349+0000","last_clean":"2024-09-15T03:18:19.174349+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:19.174349+0000","last_undegraded":"2024-09-15T03:18:19.174349+0000","last_fullsized":"2024-09-15T03:18:19.174349+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12337,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:19.177518+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:19.177518+0000","last_peered":"2024-09-15T03:18:19.177518+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:19.177518+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15607,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.173211+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:19.173211+0000","last_peered":"2024-09-15T03:18:19.173211+0000","last_clean":"2024-09-15T03:18:19.173211+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:19.173211+0000","last_undegraded":"2024-09-15T03:18:19.173211+0000","last_fullsized":"2024-09-15T03:18:19.173211+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12258,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:37.255196+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:37.255196+0000","last_peered":"2024-09-15T03:18:37.255196+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:37.255196+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1941}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12218,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.269417+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.269417+0000","last_peered":"2024-09-15T03:18:19.269417+0000","last_clean":"2024-09-15T03:18:19.269417+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.269417+0000","last_undegraded":"2024-09-15T03:18:19.269417+0000","last_fullsized":"2024-09-15T03:18:19.269417+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16186,"reported_epoch":44,"state":"peering","last_fresh":"2024-09-15T03:18:36.237236+0000","last_change":"2024-09-15T03:18:36.240398+0000","last_active":"2024-09-15T03:18:36.237209+0000","last_peered":"2024-09-15T03:18:36.147637+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:36.237236+0000","last_undegraded":"2024-09-15T03:18:36.237236+0000","last_fullsized":"2024-09-15T03:18:36.237236+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15948,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.174512+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:19.174512+0000","last_peered":"2024-09-15T03:18:19.174512+0000","last_clean":"2024-09-15T03:18:19.174512+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:19.174512+0000","last_undegraded":"2024-09-15T03:18:19.174512+0000","last_fullsized":"2024-09-15T03:18:19.174512+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16315,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.173417+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:19.173417+0000","last_peered":"2024-09-15T03:18:19.173417+0000","last_clean":"2024-09-15T03:18:19.173417+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:19.173417+0000","last_undegraded":"2024-09-15T03:18:19.173417+0000","last_fullsized":"2024-09-15T03:18:19.173417+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11997,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:19.177309+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:19.177309+0000","last_peered":"2024-09-15T03:18:19.177309+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:19.177309+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14439,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927928,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443912,"statfs":{"total":96636764160,"available":95686565888,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.58799999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914996,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456844,"statfs":{"total":96636764160,"available":95699808256,"internally_reserved":0,"allocated":368640,"data_stored":158141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926056,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445784,"statfs":{"total":96636764160,"available":95688482816,"internally_reserved":0,"allocated":552960,"data_stored":339315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.439}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.63},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.753}]}]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919608,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452232,"statfs":{"total":96636764160,"available":95695085568,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.46600000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.61099999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.75800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:39.115 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:39.115 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11986, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.236837+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:36.236837+0000', 'last_peered': '2024-09-15T03:18:36.236837+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:36.236837+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12337, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:19.177518+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:19.177518+0000', 'last_peered': '2024-09-15T03:18:19.177518+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:19.177518+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12195, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233068+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:36.233068+0000', 'last_peered': '2024-09-15T03:18:36.233068+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:36.233068+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12092, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233418+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:36.233418+0000', 'last_peered': '2024-09-15T03:18:36.233418+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:36.233418+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 12258, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:37.255196+0000', 'last_change': '2024-09-15T03:18:36.150065+0000', 'last_active': '2024-09-15T03:18:37.255196+0000', 'last_peered': '2024-09-15T03:18:37.255196+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:37.255196+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1943, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 113, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1941}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T03:18:39.116 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11966", 'reported_seq': 16186, 'reported_epoch': 44, 'state': 'peering', 'last_fresh': '2024-09-15T03:18:36.237236+0000', 'last_change': '2024-09-15T03:18:36.240398+0000', 'last_active': '2024-09-15T03:18:36.237209+0000', 'last_peered': '2024-09-15T03:18:36.147637+0000', 'last_clean': '2024-09-15T03:04:56.843739+0000', 'last_became_active': '2024-09-15T03:06:09.226229+0000', 'last_became_peered': '2024-09-15T03:06:09.226229+0000', 'last_unstale': '2024-09-15T03:18:36.237236+0000', 'last_undegraded': '2024-09-15T03:18:36.237236+0000', 'last_fullsized': '2024-09-15T03:18:36.237236+0000', 'mapping_epoch': 44, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 47862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2062, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 3], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:39.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:39.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11984, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233159+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:36.233159+0000', 'last_peered': '2024-09-15T03:18:36.233159+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:36.233159+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:39.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:39.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11790, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233221+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:36.233221+0000', 'last_peered': '2024-09-15T03:18:36.233221+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:36.233221+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:39.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:39.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11997, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:19.177309+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:19.177309+0000', 'last_peered': '2024-09-15T03:18:19.177309+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:19.177309+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:39.117 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:39.117 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12130, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.243210+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:36.243210+0000', 'last_peered': '2024-09-15T03:18:36.243210+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:36.243210+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:39.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:39.497 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:39.497 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18245/112 objects degraded (16290.179%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":842,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":6},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3777114112,"bytes_avail":382769942528,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":18245,"degraded_total":112,"degraded_ratio":162.90178571428572,"recovering_objects_per_sec":138,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:39.512 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:39.512 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:39.829 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:39.831 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:39.843 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-15T03:18:39.008420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14439,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688588,"kb_used_data":1628,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798772,"statfs":{"total":386547056640,"available":382769942528,"internally_reserved":0,"allocated":1667072,"data_stored":815638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002725"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14905,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.174349+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:19.174349+0000","last_peered":"2024-09-15T03:18:19.174349+0000","last_clean":"2024-09-15T03:18:19.174349+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:19.174349+0000","last_undegraded":"2024-09-15T03:18:19.174349+0000","last_fullsized":"2024-09-15T03:18:19.174349+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12337,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:19.177518+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:19.177518+0000","last_peered":"2024-09-15T03:18:19.177518+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:19.177518+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15607,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.173211+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:19.173211+0000","last_peered":"2024-09-15T03:18:19.173211+0000","last_clean":"2024-09-15T03:18:19.173211+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:19.173211+0000","last_undegraded":"2024-09-15T03:18:19.173211+0000","last_fullsized":"2024-09-15T03:18:19.173211+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":12258,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:37.255196+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:37.255196+0000","last_peered":"2024-09-15T03:18:37.255196+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:37.255196+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1943,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1941}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12218,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.269417+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:19.269417+0000","last_peered":"2024-09-15T03:18:19.269417+0000","last_clean":"2024-09-15T03:18:19.269417+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:19.269417+0000","last_undegraded":"2024-09-15T03:18:19.269417+0000","last_fullsized":"2024-09-15T03:18:19.269417+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16186,"reported_epoch":44,"state":"peering","last_fresh":"2024-09-15T03:18:36.237236+0000","last_change":"2024-09-15T03:18:36.240398+0000","last_active":"2024-09-15T03:18:36.237209+0000","last_peered":"2024-09-15T03:18:36.147637+0000","last_clean":"2024-09-15T03:04:56.843739+0000","last_became_active":"2024-09-15T03:06:09.226229+0000","last_became_peered":"2024-09-15T03:06:09.226229+0000","last_unstale":"2024-09-15T03:18:36.237236+0000","last_undegraded":"2024-09-15T03:18:36.237236+0000","last_fullsized":"2024-09-15T03:18:36.237236+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15948,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.174512+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:19.174512+0000","last_peered":"2024-09-15T03:18:19.174512+0000","last_clean":"2024-09-15T03:18:19.174512+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:19.174512+0000","last_undegraded":"2024-09-15T03:18:19.174512+0000","last_fullsized":"2024-09-15T03:18:19.174512+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16315,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T03:18:19.173417+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:19.173417+0000","last_peered":"2024-09-15T03:18:19.173417+0000","last_clean":"2024-09-15T03:18:19.173417+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:19.173417+0000","last_undegraded":"2024-09-15T03:18:19.173417+0000","last_fullsized":"2024-09-15T03:18:19.173417+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11997,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:19.177309+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:19.177309+0000","last_peered":"2024-09-15T03:18:19.177309+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:19.177309+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14439,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":927928,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":927547,"kb_avail":93443912,"statfs":{"total":96636764160,"available":95686565888,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":949808705},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.63400000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.58799999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914996,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456844,"statfs":{"total":96636764160,"available":95699808256,"internally_reserved":0,"allocated":368640,"data_stored":158141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926056,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445784,"statfs":{"total":96636764160,"available":95688482816,"internally_reserved":0,"allocated":552960,"data_stored":339315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.439}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.63},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.753}]}]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919608,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452232,"statfs":{"total":96636764160,"available":95695085568,"internally_reserved":0,"allocated":372736,"data_stored":159091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.46600000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.55200000000000005},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.61099999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.75800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:42.845 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:43.164 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:43.167 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:43.179 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-15T03:18:43.009366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696944,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790416,"statfs":{"total":386547056640,"available":382761385984,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002307"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":13936,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:42.259001+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:42.259001+0000","last_peered":"2024-09-15T03:18:42.259001+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:42.259001+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1102}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930220,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441620,"statfs":{"total":96636764160,"available":95684218880,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.51900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.74299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.51000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.379},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.47099999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:43.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:43.518 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:43.521 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:43.535 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-15T03:18:43.009366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696944,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790416,"statfs":{"total":386547056640,"available":382761385984,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002307"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":13936,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:42.259001+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:42.259001+0000","last_peered":"2024-09-15T03:18:42.259001+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:42.259001+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1102}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930220,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441620,"statfs":{"total":96636764160,"available":95684218880,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.51900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.74299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.51000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.379},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.47099999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:43.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:43.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11986, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.236837+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:36.236837+0000', 'last_peered': '2024-09-15T03:18:36.236837+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:36.236837+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:43.536 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:43.536 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12338, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233831+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:36.233831+0000', 'last_peered': '2024-09-15T03:18:36.233831+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:36.233831+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12195, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233068+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:36.233068+0000', 'last_peered': '2024-09-15T03:18:36.233068+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:36.233068+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12092, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233418+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:36.233418+0000', 'last_peered': '2024-09-15T03:18:36.233418+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:36.233418+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 13936, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:42.259001+0000', 'last_change': '2024-09-15T03:18:36.150065+0000', 'last_active': '2024-09-15T03:18:42.259001+0000', 'last_peered': '2024-09-15T03:18:42.259001+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:42.259001+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 952, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1102}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11984, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233159+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:36.233159+0000', 'last_peered': '2024-09-15T03:18:36.233159+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:36.233159+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11790, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233221+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:36.233221+0000', 'last_peered': '2024-09-15T03:18:36.233221+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:36.233221+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:43.537 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:43.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11998, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233239+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:36.233239+0000', 'last_peered': '2024-09-15T03:18:36.233239+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:36.233239+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:43.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:43.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12130, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.243210+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:36.243210+0000', 'last_peered': '2024-09-15T03:18:36.243210+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:36.243210+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:43.538 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:43.905 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:43.906 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17406/112 objects degraded (15541.071%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":846,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3785670656,"bytes_avail":382761385984,"bytes_total":386547056640,"degraded_objects":17406,"degraded_total":112,"degraded_ratio":155.41071428571428,"recovering_objects_per_sec":178,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1430,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:43.922 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:43.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:44.218 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:44.220 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:44.231 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-15T03:18:43.009366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3696944,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790416,"statfs":{"total":386547056640,"available":382761385984,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002307"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":13936,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:42.259001+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:42.259001+0000","last_peered":"2024-09-15T03:18:42.259001+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:42.259001+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1102}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930220,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441620,"statfs":{"total":96636764160,"available":95684218880,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.55200000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.51900000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.74299999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.51000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.379},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.47099999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:47.233 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:47.529 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:47.533 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:47.543 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-15T03:18:47.010526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688624,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798736,"statfs":{"total":386547056640,"available":382769905664,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002784"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":13936,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:42.259001+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:42.259001+0000","last_peered":"2024-09-15T03:18:42.259001+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:42.259001+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1102}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921900,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449940,"statfs":{"total":96636764160,"available":95692738560,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.46200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.51000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.379},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.47099999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:47.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:47.837 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:47.839 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:47.849 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-15T03:18:47.010526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688624,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798736,"statfs":{"total":386547056640,"available":382769905664,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002784"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":13936,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:42.259001+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:42.259001+0000","last_peered":"2024-09-15T03:18:42.259001+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:42.259001+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1102}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921900,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449940,"statfs":{"total":96636764160,"available":95692738560,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.46200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.51000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.379},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.47099999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:47.850 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11986, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.236837+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:36.236837+0000', 'last_peered': '2024-09-15T03:18:36.236837+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:36.236837+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12338, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233831+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:36.233831+0000', 'last_peered': '2024-09-15T03:18:36.233831+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:36.233831+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12195, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233068+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:36.233068+0000', 'last_peered': '2024-09-15T03:18:36.233068+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:36.233068+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12092, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233418+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:36.233418+0000', 'last_peered': '2024-09-15T03:18:36.233418+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:36.233418+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 13936, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:42.259001+0000', 'last_change': '2024-09-15T03:18:36.150065+0000', 'last_active': '2024-09-15T03:18:42.259001+0000', 'last_peered': '2024-09-15T03:18:42.259001+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:42.259001+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 952, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1102}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:47.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:47.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11984, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233159+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:36.233159+0000', 'last_peered': '2024-09-15T03:18:36.233159+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:36.233159+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:47.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:47.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11790, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233221+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:36.233221+0000', 'last_peered': '2024-09-15T03:18:36.233221+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:36.233221+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:47.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:47.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11998, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233239+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:36.233239+0000', 'last_peered': '2024-09-15T03:18:36.233239+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:36.233239+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:47.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:47.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12130, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.243210+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:36.243210+0000', 'last_peered': '2024-09-15T03:18:36.243210+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:36.243210+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:47.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:48.212 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:48.212 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17406/112 objects degraded (15541.071%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":851,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3777150976,"bytes_avail":382769905664,"bytes_total":386547056640,"degraded_objects":17406,"degraded_total":112,"degraded_ratio":155.41071428571428,"recovering_objects_per_sec":142,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1430,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:48.226 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:48.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:48.523 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:48.526 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:48.537 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-15T03:18:47.010526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688624,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798736,"statfs":{"total":386547056640,"available":382769905664,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002784"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":13936,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:42.259001+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:42.259001+0000","last_peered":"2024-09-15T03:18:42.259001+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:42.259001+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1102}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15278,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.32400000000000001},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.79200000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.755}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921900,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449940,"statfs":{"total":96636764160,"available":95692738560,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.46200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.51000000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.379},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.47099999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.63500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:51.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:51.885 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:51.887 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:51.901 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-15T03:18:51.011712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688624,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798736,"statfs":{"total":386547056640,"available":382769905664,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002345"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":15674,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:47.260148+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:47.260148+0000","last_peered":"2024-09-15T03:18:47.260148+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:47.260148+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.73999999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.76100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.443},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.80000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.47899999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921900,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449940,"statfs":{"total":96636764160,"available":95692738560,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.46200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.64400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.42499999999999999},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.55000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.625},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.67300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:51.903 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:52.232 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:52.235 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:52.247 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-15T03:18:51.011712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688624,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798736,"statfs":{"total":386547056640,"available":382769905664,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002345"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":15674,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:47.260148+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:47.260148+0000","last_peered":"2024-09-15T03:18:47.260148+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:47.260148+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.73999999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.76100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.443},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.80000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.47899999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921900,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449940,"statfs":{"total":96636764160,"available":95692738560,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.46200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.64400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.42499999999999999},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.55000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.625},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.67300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:52.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:52.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11986, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.236837+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:36.236837+0000', 'last_peered': '2024-09-15T03:18:36.236837+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:36.236837+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:52.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:52.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12338, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233831+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:36.233831+0000', 'last_peered': '2024-09-15T03:18:36.233831+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:36.233831+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 12195, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233068+0000', 'last_change': '2024-09-15T03:06:09.227197+0000', 'last_active': '2024-09-15T03:18:36.233068+0000', 'last_peered': '2024-09-15T03:18:36.233068+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:36.233068+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12092, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233418+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:36.233418+0000', 'last_peered': '2024-09-15T03:18:36.233418+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:36.233418+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11956", 'reported_seq': 15674, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:47.260148+0000', 'last_change': '2024-09-15T03:18:36.150065+0000', 'last_active': '2024-09-15T03:18:47.260148+0000', 'last_peered': '2024-09-15T03:18:47.260148+0000', 'last_clean': '2024-09-15T03:05:05.504319+0000', 'last_became_active': '2024-09-15T03:06:09.294484+0000', 'last_became_peered': '2024-09-15T03:06:09.294484+0000', 'last_unstale': '2024-09-15T03:18:47.260148+0000', 'last_undegraded': '2024-09-15T03:06:08.885250+0000', 'last_fullsized': '2024-09-15T03:06:08.841495+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11954, 'num_write_kb': 47820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1821, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -233}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11984, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233159+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:36.233159+0000', 'last_peered': '2024-09-15T03:18:36.233159+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:36.233159+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11790, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233221+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:36.233221+0000', 'last_peered': '2024-09-15T03:18:36.233221+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:36.233221+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:52.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:52.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11998, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.233239+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:36.233239+0000', 'last_peered': '2024-09-15T03:18:36.233239+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:36.233239+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:52.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:52.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12130, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:36.243210+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:36.243210+0000', 'last_peered': '2024-09-15T03:18:36.243210+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:36.243210+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:52.250 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:52.624 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:52.625 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16537/112 objects degraded (14765.179%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":855,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3777150976,"bytes_avail":382769905664,"bytes_total":386547056640,"degraded_objects":16537,"degraded_total":112,"degraded_ratio":147.65178571428572,"recovering_objects_per_sec":108,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:52.640 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:52.640 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:52.949 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:52.952 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:52.966 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-15T03:18:51.011712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3688624,"kb_used_data":1660,"kb_used_omap":14,"kb_used_meta":3686897,"kb_avail":373798736,"statfs":{"total":386547056640,"available":382769905664,"internally_reserved":0,"allocated":1699840,"data_stored":831062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3775382555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002345"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11986,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.236837+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:36.236837+0000","last_peered":"2024-09-15T03:18:36.236837+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:36.236837+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240855+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:36.240855+0000","last_peered":"2024-09-15T03:18:36.240855+0000","last_clean":"2024-09-15T03:18:36.240855+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:36.240855+0000","last_undegraded":"2024-09-15T03:18:36.240855+0000","last_fullsized":"2024-09-15T03:18:36.240855+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233831+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:36.233831+0000","last_peered":"2024-09-15T03:18:36.233831+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:36.233831+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235781+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:36.235781+0000","last_peered":"2024-09-15T03:18:36.235781+0000","last_clean":"2024-09-15T03:18:36.235781+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:36.235781+0000","last_undegraded":"2024-09-15T03:18:36.235781+0000","last_fullsized":"2024-09-15T03:18:36.235781+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":12195,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233068+0000","last_change":"2024-09-15T03:06:09.227197+0000","last_active":"2024-09-15T03:18:36.233068+0000","last_peered":"2024-09-15T03:18:36.233068+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:36.233068+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12092,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233418+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:36.233418+0000","last_peered":"2024-09-15T03:18:36.233418+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:36.233418+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":15674,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:47.260148+0000","last_change":"2024-09-15T03:18:36.150065+0000","last_active":"2024-09-15T03:18:47.260148+0000","last_peered":"2024-09-15T03:18:47.260148+0000","last_clean":"2024-09-15T03:05:05.504319+0000","last_became_active":"2024-09-15T03:06:09.294484+0000","last_became_peered":"2024-09-15T03:06:09.294484+0000","last_unstale":"2024-09-15T03:18:47.260148+0000","last_undegraded":"2024-09-15T03:06:08.885250+0000","last_fullsized":"2024-09-15T03:06:08.841495+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1821,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-233}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240582+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:36.240582+0000","last_peered":"2024-09-15T03:18:36.240582+0000","last_clean":"2024-09-15T03:18:36.240582+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:36.240582+0000","last_undegraded":"2024-09-15T03:18:36.240582+0000","last_fullsized":"2024-09-15T03:18:36.240582+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.233036+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:36.233036+0000","last_peered":"2024-09-15T03:18:36.233036+0000","last_clean":"2024-09-15T03:18:36.233036+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:36.233036+0000","last_undegraded":"2024-09-15T03:18:36.233036+0000","last_fullsized":"2024-09-15T03:18:36.233036+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16196,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T03:18:37.340382+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:37.340382+0000","last_peered":"2024-09-15T03:18:37.340382+0000","last_clean":"2024-09-15T03:18:37.340382+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:37.340382+0000","last_undegraded":"2024-09-15T03:18:37.340382+0000","last_fullsized":"2024-09-15T03:18:37.340382+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11984,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233159+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:36.233159+0000","last_peered":"2024-09-15T03:18:36.233159+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:36.233159+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15949,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.240746+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:36.240746+0000","last_peered":"2024-09-15T03:18:36.240746+0000","last_clean":"2024-09-15T03:18:36.240746+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:36.240746+0000","last_undegraded":"2024-09-15T03:18:36.240746+0000","last_fullsized":"2024-09-15T03:18:36.240746+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16316,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T03:18:36.235825+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:36.235825+0000","last_peered":"2024-09-15T03:18:36.235825+0000","last_clean":"2024-09-15T03:18:36.235825+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:36.235825+0000","last_undegraded":"2024-09-15T03:18:36.235825+0000","last_fullsized":"2024-09-15T03:18:36.235825+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11790,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233221+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:36.233221+0000","last_peered":"2024-09-15T03:18:36.233221+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:36.233221+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11998,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.233239+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:36.233239+0000","last_peered":"2024-09-15T03:18:36.233239+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:36.233239+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:36.243210+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:36.243210+0000","last_peered":"2024-09-15T03:18:36.243210+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:36.243210+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16147,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932100,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439740,"statfs":{"total":96636764160,"available":95682293760,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":3.2450000000000001,"5min":1.867,"15min":1.754},"min":{"1min":0.22600000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":34.892000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.6919999999999999,"5min":2.226,"15min":2.0720000000000001},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.185},"max":{"1min":11.006,"5min":11.006,"15min":26.143000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:17:49 2024","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":3.6869999999999998,"15min":2.5739999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":10.6,"5min":26.143000000000001,"15min":40.784999999999997},"last":0.73999999999999999},{"interface":"front","average":{"1min":2.673,"5min":2.0179999999999998,"15min":1.796},"min":{"1min":0.27900000000000003,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":20.236999999999998,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.76100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915004,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456836,"statfs":{"total":96636764160,"available":95699800064,"internally_reserved":0,"allocated":376832,"data_stored":161997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.443},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.80000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.47899999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921900,"kb_used_data":548,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449940,"statfs":{"total":96636764160,"available":95692738560,"internally_reserved":0,"allocated":561152,"data_stored":343171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.46200000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.64000000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919620,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452220,"statfs":{"total":96636764160,"available":95695073280,"internally_reserved":0,"allocated":380928,"data_stored":162947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.58399999999999996},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.64400000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.42499999999999999},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.55000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.625},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.67300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:55.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:56.283 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:56.286 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:56.298 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-15T03:18:55.012874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16850,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697008,"kb_used_data":1716,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790352,"statfs":{"total":386547056640,"available":382761320448,"internally_reserved":0,"allocated":1757184,"data_stored":853506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1572,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002347"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11988,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.281778+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:49.281778+0000","last_peered":"2024-09-15T03:18:49.281778+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:49.281778+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.294586+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:49.294586+0000","last_peered":"2024-09-15T03:18:49.294586+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:49.294586+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":13134,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:52.389604+0000","last_change":"2024-09-15T03:18:48.547106+0000","last_active":"2024-09-15T03:18:52.389604+0000","last_peered":"2024-09-15T03:18:52.389604+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:52.389604+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12094,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282847+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:49.282847+0000","last_peered":"2024-09-15T03:18:49.282847+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:49.282847+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.389490+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.389490+0000","last_peered":"2024-09-15T03:18:50.389490+0000","last_clean":"2024-09-15T03:18:50.389490+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.389490+0000","last_undegraded":"2024-09-15T03:18:50.389490+0000","last_fullsized":"2024-09-15T03:18:50.389490+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16142,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.282352+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:49.282352+0000","last_peered":"2024-09-15T03:18:49.282352+0000","last_clean":"2024-09-15T03:18:49.282352+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:49.282352+0000","last_undegraded":"2024-09-15T03:18:49.282352+0000","last_fullsized":"2024-09-15T03:18:49.282352+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.294308+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:49.294308+0000","last_peered":"2024-09-15T03:18:49.294308+0000","last_clean":"2024-09-15T03:18:49.294308+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:49.294308+0000","last_undegraded":"2024-09-15T03:18:49.294308+0000","last_fullsized":"2024-09-15T03:18:49.294308+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11986,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282708+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:49.282708+0000","last_peered":"2024-09-15T03:18:49.282708+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:49.282708+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11792,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.283289+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:49.283289+0000","last_peered":"2024-09-15T03:18:49.283289+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:49.283289+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12000,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.295058+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:49.295058+0000","last_peered":"2024-09-15T03:18:49.295058+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:49.295058+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12132,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.286185+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:49.286185+0000","last_peered":"2024-09-15T03:18:49.286185+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:49.286185+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16850,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":2.423},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":2.4849999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.47999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.625}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.90600000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.84999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930232,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441608,"statfs":{"total":96636764160,"available":95684206592,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.75700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.85799999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919636,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452204,"statfs":{"total":96636764160,"available":95695056896,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.69099999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45800000000000002},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:56.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:56.622 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:56.625 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:56.639 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-15T03:18:55.012874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16850,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697008,"kb_used_data":1716,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790352,"statfs":{"total":386547056640,"available":382761320448,"internally_reserved":0,"allocated":1757184,"data_stored":853506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1572,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002347"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11988,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.281778+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:49.281778+0000","last_peered":"2024-09-15T03:18:49.281778+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:49.281778+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.294586+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:49.294586+0000","last_peered":"2024-09-15T03:18:49.294586+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:49.294586+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":13134,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:52.389604+0000","last_change":"2024-09-15T03:18:48.547106+0000","last_active":"2024-09-15T03:18:52.389604+0000","last_peered":"2024-09-15T03:18:52.389604+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:52.389604+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12094,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282847+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:49.282847+0000","last_peered":"2024-09-15T03:18:49.282847+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:49.282847+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.389490+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.389490+0000","last_peered":"2024-09-15T03:18:50.389490+0000","last_clean":"2024-09-15T03:18:50.389490+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.389490+0000","last_undegraded":"2024-09-15T03:18:50.389490+0000","last_fullsized":"2024-09-15T03:18:50.389490+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16142,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.282352+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:49.282352+0000","last_peered":"2024-09-15T03:18:49.282352+0000","last_clean":"2024-09-15T03:18:49.282352+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:49.282352+0000","last_undegraded":"2024-09-15T03:18:49.282352+0000","last_fullsized":"2024-09-15T03:18:49.282352+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.294308+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:49.294308+0000","last_peered":"2024-09-15T03:18:49.294308+0000","last_clean":"2024-09-15T03:18:49.294308+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:49.294308+0000","last_undegraded":"2024-09-15T03:18:49.294308+0000","last_fullsized":"2024-09-15T03:18:49.294308+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11986,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282708+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:49.282708+0000","last_peered":"2024-09-15T03:18:49.282708+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:49.282708+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11792,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.283289+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:49.283289+0000","last_peered":"2024-09-15T03:18:49.283289+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:49.283289+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12000,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.295058+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:49.295058+0000","last_peered":"2024-09-15T03:18:49.295058+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:49.295058+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12132,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.286185+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:49.286185+0000","last_peered":"2024-09-15T03:18:49.286185+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:49.286185+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16850,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":2.423},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":2.4849999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.47999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.625}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.90600000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.84999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930232,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441608,"statfs":{"total":96636764160,"available":95684206592,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.75700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.85799999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919636,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452204,"statfs":{"total":96636764160,"available":95695056896,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.69099999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45800000000000002},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:18:56.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:18:56.640 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11988, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.281778+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:49.281778+0000', 'last_peered': '2024-09-15T03:18:49.281778+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:49.281778+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:56.640 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12340, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.294586+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:49.294586+0000', 'last_peered': '2024-09-15T03:18:49.294586+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:49.294586+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 13134, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:52.389604+0000', 'last_change': '2024-09-15T03:18:48.547106+0000', 'last_active': '2024-09-15T03:18:52.389604+0000', 'last_peered': '2024-09-15T03:18:52.389604+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:52.389604+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1618, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 493, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1616}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12094, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.282847+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:49.282847+0000', 'last_peered': '2024-09-15T03:18:49.282847+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:49.282847+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11986, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.282708+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:49.282708+0000', 'last_peered': '2024-09-15T03:18:49.282708+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:49.282708+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:18:56.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11792, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.283289+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:49.283289+0000', 'last_peered': '2024-09-15T03:18:49.283289+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:49.283289+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:18:56.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:18:56.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 12000, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.295058+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:49.295058+0000', 'last_peered': '2024-09-15T03:18:49.295058+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:49.295058+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:18:56.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:18:56.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12132, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.286185+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:49.286185+0000', 'last_peered': '2024-09-15T03:18:49.286185+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:49.286185+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:18:56.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15834/112 objects degraded (14137.500%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":860,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3785736192,"bytes_avail":382761320448,"bytes_total":386547056640,"degraded_objects":15834,"degraded_total":112,"degraded_ratio":141.375,"recovering_objects_per_sec":196,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1572,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:18:57.021 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:18:57.021 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:18:57.312 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:18:57.317 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:18:57.329 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2024-09-15T03:18:57.013436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16850,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701164,"kb_used_data":1716,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786196,"statfs":{"total":386547056640,"available":382757064704,"internally_reserved":0,"allocated":1757184,"data_stored":853506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-703,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":703,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002433"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11988,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.281778+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:49.281778+0000","last_peered":"2024-09-15T03:18:49.281778+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:49.281778+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.294586+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:49.294586+0000","last_peered":"2024-09-15T03:18:49.294586+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:49.294586+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":13134,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:52.389604+0000","last_change":"2024-09-15T03:18:48.547106+0000","last_active":"2024-09-15T03:18:52.389604+0000","last_peered":"2024-09-15T03:18:52.389604+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:52.389604+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12094,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282847+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:49.282847+0000","last_peered":"2024-09-15T03:18:49.282847+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:49.282847+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.389490+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.389490+0000","last_peered":"2024-09-15T03:18:50.389490+0000","last_clean":"2024-09-15T03:18:50.389490+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.389490+0000","last_undegraded":"2024-09-15T03:18:50.389490+0000","last_fullsized":"2024-09-15T03:18:50.389490+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16142,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.282352+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:49.282352+0000","last_peered":"2024-09-15T03:18:49.282352+0000","last_clean":"2024-09-15T03:18:49.282352+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:49.282352+0000","last_undegraded":"2024-09-15T03:18:49.282352+0000","last_fullsized":"2024-09-15T03:18:49.282352+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.294308+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:49.294308+0000","last_peered":"2024-09-15T03:18:49.294308+0000","last_clean":"2024-09-15T03:18:49.294308+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:49.294308+0000","last_undegraded":"2024-09-15T03:18:49.294308+0000","last_fullsized":"2024-09-15T03:18:49.294308+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11986,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282708+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:49.282708+0000","last_peered":"2024-09-15T03:18:49.282708+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:49.282708+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11792,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.283289+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:49.283289+0000","last_peered":"2024-09-15T03:18:49.283289+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:49.283289+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12000,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.295058+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:49.295058+0000","last_peered":"2024-09-15T03:18:49.295058+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:49.295058+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12132,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.286185+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:49.286185+0000","last_peered":"2024-09-15T03:18:49.286185+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:49.286185+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16850,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54800000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":2.423},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":2.4849999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.47999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0670000000000002,"5min":2.8170000000000002,"15min":2.2109999999999999},"min":{"1min":0.192,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":20.701000000000001,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.5329999999999999,"15min":1.4790000000000001},"min":{"1min":0.27400000000000002,"5min":0.192,"15min":0.192},"max":{"1min":7.75,"5min":19.277000000000001,"15min":19.329000000000001},"last":0.625}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.90600000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":2.04,"15min":1.819},"min":{"1min":0.29499999999999998,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":9.2240000000000002,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.8799999999999999,"5min":2.3250000000000002,"15min":1.948},"min":{"1min":0.34000000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0940000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.84999999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934388,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437452,"statfs":{"total":96636764160,"available":95679950848,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.64700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.621}]}]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919636,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452204,"statfs":{"total":96636764160,"available":95695056896,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.69099999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:17:50 2024","interfaces":[{"interface":"back","average":{"1min":4.1669999999999998,"5min":2.6640000000000001,"15min":2.383},"min":{"1min":0.22700000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":18.474,"5min":30.562999999999999,"15min":30.802},"last":0.45800000000000002},{"interface":"front","average":{"1min":2.355,"5min":2.2509999999999999,"15min":1.7350000000000001},"min":{"1min":0.23999999999999999,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":10.321999999999999,"5min":19.722000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.75900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:00.331 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:00.629 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:00.632 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:00.642 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-15T03:18:59.014112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17689,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701164,"kb_used_data":1716,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786196,"statfs":{"total":386547056640,"available":382757064704,"internally_reserved":0,"allocated":1757184,"data_stored":853506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1542,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.735306"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11988,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.281778+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:49.281778+0000","last_peered":"2024-09-15T03:18:49.281778+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:49.281778+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.294586+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:49.294586+0000","last_peered":"2024-09-15T03:18:49.294586+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:49.294586+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":14812,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:57.393698+0000","last_change":"2024-09-15T03:18:48.547106+0000","last_active":"2024-09-15T03:18:57.393698+0000","last_peered":"2024-09-15T03:18:57.393698+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:57.393698+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1332,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12094,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282847+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:49.282847+0000","last_peered":"2024-09-15T03:18:49.282847+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:49.282847+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.389490+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.389490+0000","last_peered":"2024-09-15T03:18:50.389490+0000","last_clean":"2024-09-15T03:18:50.389490+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.389490+0000","last_undegraded":"2024-09-15T03:18:50.389490+0000","last_fullsized":"2024-09-15T03:18:50.389490+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16142,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.282352+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:49.282352+0000","last_peered":"2024-09-15T03:18:49.282352+0000","last_clean":"2024-09-15T03:18:49.282352+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:49.282352+0000","last_undegraded":"2024-09-15T03:18:49.282352+0000","last_fullsized":"2024-09-15T03:18:49.282352+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.294308+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:49.294308+0000","last_peered":"2024-09-15T03:18:49.294308+0000","last_clean":"2024-09-15T03:18:49.294308+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:49.294308+0000","last_undegraded":"2024-09-15T03:18:49.294308+0000","last_fullsized":"2024-09-15T03:18:49.294308+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11986,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282708+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:49.282708+0000","last_peered":"2024-09-15T03:18:49.282708+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:49.282708+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11792,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.283289+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:49.283289+0000","last_peered":"2024-09-15T03:18:49.283289+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:49.283289+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12000,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.295058+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:49.295058+0000","last_peered":"2024-09-15T03:18:49.295058+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:49.295058+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12132,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.286185+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:49.286185+0000","last_peered":"2024-09-15T03:18:49.286185+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:49.286185+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17689,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.41299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.67000000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.625}]}]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.443},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.59199999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.35999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934388,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437452,"statfs":{"total":96636764160,"available":95679950848,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.64700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.621}]}]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919636,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452204,"statfs":{"total":96636764160,"available":95695056896,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.79700000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":1.361}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:00.644 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:00.940 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:00.942 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:00.954 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-15T03:18:59.014112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17689,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701164,"kb_used_data":1716,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786196,"statfs":{"total":386547056640,"available":382757064704,"internally_reserved":0,"allocated":1757184,"data_stored":853506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1542,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.735306"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11988,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.281778+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:49.281778+0000","last_peered":"2024-09-15T03:18:49.281778+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:49.281778+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.294586+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:49.294586+0000","last_peered":"2024-09-15T03:18:49.294586+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:49.294586+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":14812,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:57.393698+0000","last_change":"2024-09-15T03:18:48.547106+0000","last_active":"2024-09-15T03:18:57.393698+0000","last_peered":"2024-09-15T03:18:57.393698+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:57.393698+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1332,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12094,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282847+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:49.282847+0000","last_peered":"2024-09-15T03:18:49.282847+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:49.282847+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.389490+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.389490+0000","last_peered":"2024-09-15T03:18:50.389490+0000","last_clean":"2024-09-15T03:18:50.389490+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.389490+0000","last_undegraded":"2024-09-15T03:18:50.389490+0000","last_fullsized":"2024-09-15T03:18:50.389490+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16142,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.282352+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:49.282352+0000","last_peered":"2024-09-15T03:18:49.282352+0000","last_clean":"2024-09-15T03:18:49.282352+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:49.282352+0000","last_undegraded":"2024-09-15T03:18:49.282352+0000","last_fullsized":"2024-09-15T03:18:49.282352+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.294308+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:49.294308+0000","last_peered":"2024-09-15T03:18:49.294308+0000","last_clean":"2024-09-15T03:18:49.294308+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:49.294308+0000","last_undegraded":"2024-09-15T03:18:49.294308+0000","last_fullsized":"2024-09-15T03:18:49.294308+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11986,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282708+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:49.282708+0000","last_peered":"2024-09-15T03:18:49.282708+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:49.282708+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11792,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.283289+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:49.283289+0000","last_peered":"2024-09-15T03:18:49.283289+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:49.283289+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12000,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.295058+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:49.295058+0000","last_peered":"2024-09-15T03:18:49.295058+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:49.295058+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12132,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.286185+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:49.286185+0000","last_peered":"2024-09-15T03:18:49.286185+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:49.286185+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17689,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.41299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.67000000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.625}]}]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.443},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.59199999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.35999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934388,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437452,"statfs":{"total":96636764160,"available":95679950848,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.64700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.621}]}]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919636,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452204,"statfs":{"total":96636764160,"available":95695056896,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.79700000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":1.361}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:00.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:00.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11988, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.281778+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:49.281778+0000', 'last_peered': '2024-09-15T03:18:49.281778+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:49.281778+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:00.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:00.956 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12340, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.294586+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:49.294586+0000', 'last_peered': '2024-09-15T03:18:49.294586+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:49.294586+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:00.956 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:19:00.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 14812, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:57.393698+0000', 'last_change': '2024-09-15T03:18:48.547106+0000', 'last_active': '2024-09-15T03:18:57.393698+0000', 'last_peered': '2024-09-15T03:18:57.393698+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:18:57.393698+0000', 'last_undegraded': '2024-09-15T03:06:08.898236+0000', 'last_fullsized': '2024-09-15T03:06:08.835570+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 779, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1332, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -777}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:00.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:00.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12094, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.282847+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:49.282847+0000', 'last_peered': '2024-09-15T03:18:49.282847+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:49.282847+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:00.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:00.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11986, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.282708+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:49.282708+0000', 'last_peered': '2024-09-15T03:18:49.282708+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:49.282708+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:00.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:00.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11792, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.283289+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:49.283289+0000', 'last_peered': '2024-09-15T03:18:49.283289+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:49.283289+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:00.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:19:00.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 12000, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.295058+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:49.295058+0000', 'last_peered': '2024-09-15T03:18:49.295058+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:49.295058+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:00.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:19:00.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12132, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:49.286185+0000', 'last_change': '2024-09-15T03:06:09.330175+0000', 'last_active': '2024-09-15T03:18:49.286185+0000', 'last_peered': '2024-09-15T03:18:49.286185+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:18:49.286185+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2061, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:19:00.959 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:01.327 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:01.327 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14995/112 objects degraded (13388.393%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":864,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3789991936,"bytes_avail":382757064704,"bytes_total":386547056640,"degraded_objects":14995,"degraded_total":112,"degraded_ratio":133.88392857142858,"recovering_objects_per_sec":143,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1542,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:01.342 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:01.343 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:01.633 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:01.636 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:01.650 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-15T03:19:01.014885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17689,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701164,"kb_used_data":1716,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786196,"statfs":{"total":386547056640,"available":382757064704,"internally_reserved":0,"allocated":1757184,"data_stored":853506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1542,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.726771"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11988,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.281778+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:49.281778+0000","last_peered":"2024-09-15T03:18:49.281778+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:49.281778+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.294586+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:49.294586+0000","last_peered":"2024-09-15T03:18:49.294586+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:49.294586+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":14812,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:57.393698+0000","last_change":"2024-09-15T03:18:48.547106+0000","last_active":"2024-09-15T03:18:57.393698+0000","last_peered":"2024-09-15T03:18:57.393698+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:18:57.393698+0000","last_undegraded":"2024-09-15T03:06:08.898236+0000","last_fullsized":"2024-09-15T03:06:08.835570+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":779,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1332,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-777}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12094,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282847+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:49.282847+0000","last_peered":"2024-09-15T03:18:49.282847+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:49.282847+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16158,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.389490+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.389490+0000","last_peered":"2024-09-15T03:18:50.389490+0000","last_clean":"2024-09-15T03:18:50.389490+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.389490+0000","last_undegraded":"2024-09-15T03:18:50.389490+0000","last_fullsized":"2024-09-15T03:18:50.389490+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16142,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.282352+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:49.282352+0000","last_peered":"2024-09-15T03:18:49.282352+0000","last_clean":"2024-09-15T03:18:49.282352+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:49.282352+0000","last_undegraded":"2024-09-15T03:18:49.282352+0000","last_fullsized":"2024-09-15T03:18:49.282352+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.294308+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:49.294308+0000","last_peered":"2024-09-15T03:18:49.294308+0000","last_clean":"2024-09-15T03:18:49.294308+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:49.294308+0000","last_undegraded":"2024-09-15T03:18:49.294308+0000","last_fullsized":"2024-09-15T03:18:49.294308+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11986,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.282708+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:49.282708+0000","last_peered":"2024-09-15T03:18:49.282708+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:49.282708+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11792,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.283289+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:49.283289+0000","last_peered":"2024-09-15T03:18:49.283289+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:49.283289+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12000,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.295058+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:49.295058+0000","last_peered":"2024-09-15T03:18:49.295058+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:49.295058+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12132,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:49.286185+0000","last_change":"2024-09-15T03:06:09.330175+0000","last_active":"2024-09-15T03:18:49.286185+0000","last_peered":"2024-09-15T03:18:49.286185+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:18:49.286185+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2061,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2057}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17689,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.41299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.67000000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.625}]}]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.443},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.46600000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.59199999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.35999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934388,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437452,"statfs":{"total":96636764160,"available":95679950848,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.36699999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.64700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.621}]}]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919636,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452204,"statfs":{"total":96636764160,"available":95695056896,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.79700000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.998}]},{"osd":3,"last update":"Sun Sep 15 03:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":1.351,"15min":1.4019999999999999},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.6500000000000004,"5min":15.114000000000001,"15min":20.948},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.835,"15min":1.587},"min":{"1min":0.316,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":12.644,"5min":16.562000000000001,"15min":30.013000000000002},"last":1.361}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:04.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:04.967 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:04.970 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:04.984 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-15T03:19:03.015635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692848,"kb_used_data":1724,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794512,"statfs":{"total":386547056640,"available":382765580288,"internally_reserved":0,"allocated":1765376,"data_stored":857309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002761"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11989,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.293596+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:50.293596+0000","last_peered":"2024-09-15T03:18:50.293596+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:50.293596+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12341,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.294625+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:50.294625+0000","last_peered":"2024-09-15T03:18:50.294625+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:50.294625+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16374,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-15T03:19:01.842588+0000","last_change":"2024-09-15T03:19:01.848107+0000","last_active":"2024-09-15T03:19:01.842557+0000","last_peered":"2024-09-15T03:19:01.810951+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:19:01.842588+0000","last_undegraded":"2024-09-15T03:19:01.842588+0000","last_fullsized":"2024-09-15T03:19:01.842588+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12095,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291632+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:50.291632+0000","last_peered":"2024-09-15T03:18:50.291632+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:50.291632+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16159,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.390639+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.390639+0000","last_peered":"2024-09-15T03:18:50.390639+0000","last_clean":"2024-09-15T03:18:50.390639+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.390639+0000","last_undegraded":"2024-09-15T03:18:50.390639+0000","last_fullsized":"2024-09-15T03:18:50.390639+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16143,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.291126+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:50.291126+0000","last_peered":"2024-09-15T03:18:50.291126+0000","last_clean":"2024-09-15T03:18:50.291126+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:50.291126+0000","last_undegraded":"2024-09-15T03:18:50.291126+0000","last_fullsized":"2024-09-15T03:18:50.291126+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16199,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.295746+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:50.295746+0000","last_peered":"2024-09-15T03:18:50.295746+0000","last_clean":"2024-09-15T03:18:50.295746+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:50.295746+0000","last_undegraded":"2024-09-15T03:18:50.295746+0000","last_fullsized":"2024-09-15T03:18:50.295746+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11987,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291424+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:50.291424+0000","last_peered":"2024-09-15T03:18:50.291424+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:50.291424+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291752+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:50.291752+0000","last_peered":"2024-09-15T03:18:50.291752+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:50.291752+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12001,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.292765+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:50.292765+0000","last_peered":"2024-09-15T03:18:50.292765+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:50.292765+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12278,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:02.260410+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:02.260410+0000","last_peered":"2024-09-15T03:19:02.260410+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:02.260410+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1987}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.26600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.372},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.495}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.63900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.60199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.73099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926068,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445772,"statfs":{"total":96636764160,"available":95688470528,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.377},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.59999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.63200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919640,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452200,"statfs":{"total":96636764160,"available":95695052800,"internally_reserved":0,"allocated":405504,"data_stored":174409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.318},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.39600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:04.985 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:05.311 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:05.314 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:05.324 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-15T03:19:05.016123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692848,"kb_used_data":1724,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794512,"statfs":{"total":386547056640,"available":382765580288,"internally_reserved":0,"allocated":1765376,"data_stored":857309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002686"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11989,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.293596+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:50.293596+0000","last_peered":"2024-09-15T03:18:50.293596+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:50.293596+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12341,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.294625+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:50.294625+0000","last_peered":"2024-09-15T03:18:50.294625+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:50.294625+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16374,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-15T03:19:01.842588+0000","last_change":"2024-09-15T03:19:01.848107+0000","last_active":"2024-09-15T03:19:01.842557+0000","last_peered":"2024-09-15T03:19:01.810951+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:19:01.842588+0000","last_undegraded":"2024-09-15T03:19:01.842588+0000","last_fullsized":"2024-09-15T03:19:01.842588+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12095,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291632+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:50.291632+0000","last_peered":"2024-09-15T03:18:50.291632+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:50.291632+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16159,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.390639+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.390639+0000","last_peered":"2024-09-15T03:18:50.390639+0000","last_clean":"2024-09-15T03:18:50.390639+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.390639+0000","last_undegraded":"2024-09-15T03:18:50.390639+0000","last_fullsized":"2024-09-15T03:18:50.390639+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16143,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.291126+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:50.291126+0000","last_peered":"2024-09-15T03:18:50.291126+0000","last_clean":"2024-09-15T03:18:50.291126+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:50.291126+0000","last_undegraded":"2024-09-15T03:18:50.291126+0000","last_fullsized":"2024-09-15T03:18:50.291126+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16199,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.295746+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:50.295746+0000","last_peered":"2024-09-15T03:18:50.295746+0000","last_clean":"2024-09-15T03:18:50.295746+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:50.295746+0000","last_undegraded":"2024-09-15T03:18:50.295746+0000","last_fullsized":"2024-09-15T03:18:50.295746+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11987,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291424+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:50.291424+0000","last_peered":"2024-09-15T03:18:50.291424+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:50.291424+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291752+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:50.291752+0000","last_peered":"2024-09-15T03:18:50.291752+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:50.291752+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12001,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.292765+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:50.292765+0000","last_peered":"2024-09-15T03:18:50.292765+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:50.292765+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12278,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:02.260410+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:02.260410+0000","last_peered":"2024-09-15T03:19:02.260410+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:02.260410+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1987}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.26600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.372},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.495}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.63900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.60199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.73099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926068,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445772,"statfs":{"total":96636764160,"available":95688470528,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.377},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.59999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.63200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919640,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452200,"statfs":{"total":96636764160,"available":95695052800,"internally_reserved":0,"allocated":405504,"data_stored":174409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.318},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.39600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11989, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:50.293596+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:18:50.293596+0000', 'last_peered': '2024-09-15T03:18:50.293596+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:18:50.293596+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12341, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:50.294625+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:18:50.294625+0000', 'last_peered': '2024-09-15T03:18:50.294625+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:18:50.294625+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12079", 'reported_seq': 16374, 'reported_epoch': 48, 'state': 'peering', 'last_fresh': '2024-09-15T03:19:01.842588+0000', 'last_change': '2024-09-15T03:19:01.848107+0000', 'last_active': '2024-09-15T03:19:01.842557+0000', 'last_peered': '2024-09-15T03:19:01.810951+0000', 'last_clean': '2024-09-15T03:05:04.507511+0000', 'last_became_active': '2024-09-15T03:06:09.219231+0000', 'last_became_peered': '2024-09-15T03:06:09.219231+0000', 'last_unstale': '2024-09-15T03:19:01.842588+0000', 'last_undegraded': '2024-09-15T03:19:01.842588+0000', 'last_fullsized': '2024-09-15T03:19:01.842588+0000', 'mapping_epoch': 48, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12077, 'num_write_kb': 48317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2111, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 1, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12095, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:50.291632+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:18:50.291632+0000', 'last_peered': '2024-09-15T03:18:50.291632+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:18:50.291632+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:05.326 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:05.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11987, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:50.291424+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:18:50.291424+0000', 'last_peered': '2024-09-15T03:18:50.291424+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:18:50.291424+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:05.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:05.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11793, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:50.291752+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:18:50.291752+0000', 'last_peered': '2024-09-15T03:18:50.291752+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:18:50.291752+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:05.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:19:05.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 12001, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:18:50.292765+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:18:50.292765+0000', 'last_peered': '2024-09-15T03:18:50.292765+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:18:50.292765+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:05.327 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:19:05.327 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 12278, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:02.260410+0000', 'last_change': '2024-09-15T03:19:01.812984+0000', 'last_active': '2024-09-15T03:19:02.260410+0000', 'last_peered': '2024-09-15T03:19:02.260410+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:19:02.260410+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 97, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1987}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:19:05.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:05.699 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:05.700 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14144/112 objects degraded (12628.571%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":868,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3781476352,"bytes_avail":382765580288,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":14144,"degraded_total":112,"degraded_ratio":126.28571428571429,"recovering_objects_per_sec":211,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1690,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:05.718 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:05.719 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:06.019 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:06.022 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:06.033 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-15T03:19:05.016123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692848,"kb_used_data":1724,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794512,"statfs":{"total":386547056640,"available":382765580288,"internally_reserved":0,"allocated":1765376,"data_stored":857309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1690,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002686"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11989,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.293596+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:18:50.293596+0000","last_peered":"2024-09-15T03:18:50.293596+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:18:50.293596+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14908,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281239+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:18:49.281239+0000","last_peered":"2024-09-15T03:18:49.281239+0000","last_clean":"2024-09-15T03:18:49.281239+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:18:49.281239+0000","last_undegraded":"2024-09-15T03:18:49.281239+0000","last_fullsized":"2024-09-15T03:18:49.281239+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12341,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.294625+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:18:50.294625+0000","last_peered":"2024-09-15T03:18:50.294625+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:18:50.294625+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280854+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:18:49.280854+0000","last_peered":"2024-09-15T03:18:49.280854+0000","last_clean":"2024-09-15T03:18:49.280854+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:18:49.280854+0000","last_undegraded":"2024-09-15T03:18:49.280854+0000","last_fullsized":"2024-09-15T03:18:49.280854+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16374,"reported_epoch":48,"state":"peering","last_fresh":"2024-09-15T03:19:01.842588+0000","last_change":"2024-09-15T03:19:01.848107+0000","last_active":"2024-09-15T03:19:01.842557+0000","last_peered":"2024-09-15T03:19:01.810951+0000","last_clean":"2024-09-15T03:05:04.507511+0000","last_became_active":"2024-09-15T03:06:09.219231+0000","last_became_peered":"2024-09-15T03:06:09.219231+0000","last_unstale":"2024-09-15T03:19:01.842588+0000","last_undegraded":"2024-09-15T03:19:01.842588+0000","last_fullsized":"2024-09-15T03:19:01.842588+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12095,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291632+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:18:50.291632+0000","last_peered":"2024-09-15T03:18:50.291632+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:18:50.291632+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16159,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.390639+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:18:50.390639+0000","last_peered":"2024-09-15T03:18:50.390639+0000","last_clean":"2024-09-15T03:18:50.390639+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:18:50.390639+0000","last_undegraded":"2024-09-15T03:18:50.390639+0000","last_fullsized":"2024-09-15T03:18:50.390639+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.280961+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:18:49.280961+0000","last_peered":"2024-09-15T03:18:49.280961+0000","last_clean":"2024-09-15T03:18:49.280961+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:18:49.280961+0000","last_undegraded":"2024-09-15T03:18:49.280961+0000","last_fullsized":"2024-09-15T03:18:49.280961+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16143,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.291126+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:18:50.291126+0000","last_peered":"2024-09-15T03:18:50.291126+0000","last_clean":"2024-09-15T03:18:50.291126+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:18:50.291126+0000","last_undegraded":"2024-09-15T03:18:50.291126+0000","last_fullsized":"2024-09-15T03:18:50.291126+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16199,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T03:18:50.295746+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:18:50.295746+0000","last_peered":"2024-09-15T03:18:50.295746+0000","last_clean":"2024-09-15T03:18:50.295746+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:18:50.295746+0000","last_undegraded":"2024-09-15T03:18:50.295746+0000","last_fullsized":"2024-09-15T03:18:50.295746+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11987,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291424+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:18:50.291424+0000","last_peered":"2024-09-15T03:18:50.291424+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:18:50.291424+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15951,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281303+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:18:49.281303+0000","last_peered":"2024-09-15T03:18:49.281303+0000","last_clean":"2024-09-15T03:18:49.281303+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:18:49.281303+0000","last_undegraded":"2024-09-15T03:18:49.281303+0000","last_fullsized":"2024-09-15T03:18:49.281303+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16318,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T03:18:49.281054+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:18:49.281054+0000","last_peered":"2024-09-15T03:18:49.281054+0000","last_clean":"2024-09-15T03:18:49.281054+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:18:49.281054+0000","last_undegraded":"2024-09-15T03:18:49.281054+0000","last_fullsized":"2024-09-15T03:18:49.281054+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11793,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.291752+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:18:50.291752+0000","last_peered":"2024-09-15T03:18:50.291752+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:18:50.291752+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12001,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:18:50.292765+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:18:50.292765+0000","last_peered":"2024-09-15T03:18:50.292765+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:18:50.292765+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12278,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:02.260410+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:02.260410+0000","last_peered":"2024-09-15T03:19:02.260410+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:02.260410+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1987}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":188416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932116,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439724,"statfs":{"total":96636764160,"available":95682277376,"internally_reserved":0,"allocated":397312,"data_stored":170606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.26600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.372},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56699999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.495}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":915024,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":914622,"kb_avail":93456816,"statfs":{"total":96636764160,"available":95699779584,"internally_reserved":0,"allocated":393216,"data_stored":169656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936573272},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.63900000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.60199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.73099999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926068,"kb_used_data":556,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445772,"statfs":{"total":96636764160,"available":95688470528,"internally_reserved":0,"allocated":569344,"data_stored":342638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.377},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.59999999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.63200000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919640,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452200,"statfs":{"total":96636764160,"available":95695052800,"internally_reserved":0,"allocated":405504,"data_stored":174409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.318},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.39600000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":176128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:09.035 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:09.366 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:09.369 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:09.380 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-15T03:19:09.017331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19371,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701212,"kb_used_data":1772,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786148,"statfs":{"total":386547056640,"available":382757015552,"internally_reserved":0,"allocated":1814528,"data_stored":875654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1682,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002445"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11990,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.847061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:01.847061+0000","last_peered":"2024-09-15T03:19:01.847061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:01.847061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14910,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840290+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:01.840290+0000","last_peered":"2024-09-15T03:19:01.840290+0000","last_clean":"2024-09-15T03:19:01.840290+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:01.840290+0000","last_undegraded":"2024-09-15T03:19:01.840290+0000","last_fullsized":"2024-09-15T03:19:01.840290+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.843273+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:01.843273+0000","last_peered":"2024-09-15T03:19:01.843273+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:01.843273+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15612,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839339+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:01.839339+0000","last_peered":"2024-09-15T03:19:01.839339+0000","last_clean":"2024-09-15T03:19:01.839339+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:01.839339+0000","last_undegraded":"2024-09-15T03:19:01.839339+0000","last_fullsized":"2024-09-15T03:19:01.839339+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T03:19:02.961706+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:02.961706+0000","last_peered":"2024-09-15T03:19:02.961706+0000","last_clean":"2024-09-15T03:19:02.961706+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:02.961706+0000","last_undegraded":"2024-09-15T03:19:02.961706+0000","last_fullsized":"2024-09-15T03:19:02.961706+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12096,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.845732+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:01.845732+0000","last_peered":"2024-09-15T03:19:01.845732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:01.845732+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.846508+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:01.846508+0000","last_peered":"2024-09-15T03:19:01.846508+0000","last_clean":"2024-09-15T03:19:01.846508+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:01.846508+0000","last_undegraded":"2024-09-15T03:19:01.846508+0000","last_fullsized":"2024-09-15T03:19:01.846508+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12223,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840293+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:01.840293+0000","last_peered":"2024-09-15T03:19:01.840293+0000","last_clean":"2024-09-15T03:19:01.840293+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:01.840293+0000","last_undegraded":"2024-09-15T03:19:01.840293+0000","last_fullsized":"2024-09-15T03:19:01.840293+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843855+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:01.843855+0000","last_peered":"2024-09-15T03:19:01.843855+0000","last_clean":"2024-09-15T03:19:01.843855+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:01.843855+0000","last_undegraded":"2024-09-15T03:19:01.843855+0000","last_fullsized":"2024-09-15T03:19:01.843855+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843859+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:01.843859+0000","last_peered":"2024-09-15T03:19:01.843859+0000","last_clean":"2024-09-15T03:19:01.843859+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:01.843859+0000","last_undegraded":"2024-09-15T03:19:01.843859+0000","last_fullsized":"2024-09-15T03:19:01.843859+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11988,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.844639+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:01.844639+0000","last_peered":"2024-09-15T03:19:01.844639+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:01.844639+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840343+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:01.840343+0000","last_peered":"2024-09-15T03:19:01.840343+0000","last_clean":"2024-09-15T03:19:01.840343+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:01.840343+0000","last_undegraded":"2024-09-15T03:19:01.840343+0000","last_fullsized":"2024-09-15T03:19:01.840343+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16320,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839511+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:01.839511+0000","last_peered":"2024-09-15T03:19:01.839511+0000","last_clean":"2024-09-15T03:19:01.839511+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:01.839511+0000","last_undegraded":"2024-09-15T03:19:01.839511+0000","last_fullsized":"2024-09-15T03:19:01.839511+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11794,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841036+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:01.841036+0000","last_peered":"2024-09-15T03:19:01.841036+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:01.841036+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12002,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841227+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:19:01.841227+0000","last_peered":"2024-09-15T03:19:01.841227+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:01.841227+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":13941,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:07.264158+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:07.264158+0000","last_peered":"2024-09-15T03:19:07.264158+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:07.264158+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1156}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19371,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932128,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439712,"statfs":{"total":96636764160,"available":95682265088,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919200,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452640,"statfs":{"total":96636764160,"available":95695503360,"internally_reserved":0,"allocated":409600,"data_stored":177241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.39600000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.53200000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930236,"kb_used_data":564,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441604,"statfs":{"total":96636764160,"available":95684202496,"internally_reserved":0,"allocated":577536,"data_stored":342031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.441},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.39800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919648,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452192,"statfs":{"total":96636764160,"available":95695044608,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.627},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.441}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:09.381 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:09.685 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:09.689 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:09.699 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-15T03:19:09.017331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19371,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701212,"kb_used_data":1772,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786148,"statfs":{"total":386547056640,"available":382757015552,"internally_reserved":0,"allocated":1814528,"data_stored":875654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1682,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002445"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11990,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.847061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:01.847061+0000","last_peered":"2024-09-15T03:19:01.847061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:01.847061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14910,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840290+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:01.840290+0000","last_peered":"2024-09-15T03:19:01.840290+0000","last_clean":"2024-09-15T03:19:01.840290+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:01.840290+0000","last_undegraded":"2024-09-15T03:19:01.840290+0000","last_fullsized":"2024-09-15T03:19:01.840290+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.843273+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:01.843273+0000","last_peered":"2024-09-15T03:19:01.843273+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:01.843273+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15612,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839339+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:01.839339+0000","last_peered":"2024-09-15T03:19:01.839339+0000","last_clean":"2024-09-15T03:19:01.839339+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:01.839339+0000","last_undegraded":"2024-09-15T03:19:01.839339+0000","last_fullsized":"2024-09-15T03:19:01.839339+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T03:19:02.961706+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:02.961706+0000","last_peered":"2024-09-15T03:19:02.961706+0000","last_clean":"2024-09-15T03:19:02.961706+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:02.961706+0000","last_undegraded":"2024-09-15T03:19:02.961706+0000","last_fullsized":"2024-09-15T03:19:02.961706+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12096,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.845732+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:01.845732+0000","last_peered":"2024-09-15T03:19:01.845732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:01.845732+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.846508+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:01.846508+0000","last_peered":"2024-09-15T03:19:01.846508+0000","last_clean":"2024-09-15T03:19:01.846508+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:01.846508+0000","last_undegraded":"2024-09-15T03:19:01.846508+0000","last_fullsized":"2024-09-15T03:19:01.846508+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12223,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840293+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:01.840293+0000","last_peered":"2024-09-15T03:19:01.840293+0000","last_clean":"2024-09-15T03:19:01.840293+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:01.840293+0000","last_undegraded":"2024-09-15T03:19:01.840293+0000","last_fullsized":"2024-09-15T03:19:01.840293+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843855+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:01.843855+0000","last_peered":"2024-09-15T03:19:01.843855+0000","last_clean":"2024-09-15T03:19:01.843855+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:01.843855+0000","last_undegraded":"2024-09-15T03:19:01.843855+0000","last_fullsized":"2024-09-15T03:19:01.843855+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843859+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:01.843859+0000","last_peered":"2024-09-15T03:19:01.843859+0000","last_clean":"2024-09-15T03:19:01.843859+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:01.843859+0000","last_undegraded":"2024-09-15T03:19:01.843859+0000","last_fullsized":"2024-09-15T03:19:01.843859+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11988,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.844639+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:01.844639+0000","last_peered":"2024-09-15T03:19:01.844639+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:01.844639+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840343+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:01.840343+0000","last_peered":"2024-09-15T03:19:01.840343+0000","last_clean":"2024-09-15T03:19:01.840343+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:01.840343+0000","last_undegraded":"2024-09-15T03:19:01.840343+0000","last_fullsized":"2024-09-15T03:19:01.840343+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16320,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839511+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:01.839511+0000","last_peered":"2024-09-15T03:19:01.839511+0000","last_clean":"2024-09-15T03:19:01.839511+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:01.839511+0000","last_undegraded":"2024-09-15T03:19:01.839511+0000","last_fullsized":"2024-09-15T03:19:01.839511+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11794,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841036+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:01.841036+0000","last_peered":"2024-09-15T03:19:01.841036+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:01.841036+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12002,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841227+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:19:01.841227+0000","last_peered":"2024-09-15T03:19:01.841227+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:01.841227+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":13941,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:07.264158+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:07.264158+0000","last_peered":"2024-09-15T03:19:07.264158+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:07.264158+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1156}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19371,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932128,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439712,"statfs":{"total":96636764160,"available":95682265088,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919200,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452640,"statfs":{"total":96636764160,"available":95695503360,"internally_reserved":0,"allocated":409600,"data_stored":177241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.39600000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.53200000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930236,"kb_used_data":564,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441604,"statfs":{"total":96636764160,"available":95684202496,"internally_reserved":0,"allocated":577536,"data_stored":342031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.441},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.39800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919648,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452192,"statfs":{"total":96636764160,"available":95695044608,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.627},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.441}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:09.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11990, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.847061+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:01.847061+0000', 'last_peered': '2024-09-15T03:19:01.847061+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:01.847061+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12342, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.843273+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:19:01.843273+0000', 'last_peered': '2024-09-15T03:19:01.843273+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:01.843273+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12096, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.845732+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:01.845732+0000', 'last_peered': '2024-09-15T03:19:01.845732+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:01.845732+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11988, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.844639+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:19:01.844639+0000', 'last_peered': '2024-09-15T03:19:01.844639+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:01.844639+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:09.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11794, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.841036+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:01.841036+0000', 'last_peered': '2024-09-15T03:19:01.841036+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:01.841036+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:09.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:19:09.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 12002, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.841227+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:19:01.841227+0000', 'last_peered': '2024-09-15T03:19:01.841227+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:19:01.841227+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:09.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:19:09.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 13941, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:07.264158+0000', 'last_change': '2024-09-15T03:19:01.812984+0000', 'last_active': '2024-09-15T03:19:07.264158+0000', 'last_peered': '2024-09-15T03:19:07.264158+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:19:07.264158+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1158, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 928, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1156}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:19:09.702 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:10.073 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:10.073 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13313/112 objects degraded (11886.607%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":873,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3790041088,"bytes_avail":382757015552,"bytes_total":386547056640,"degraded_objects":13313,"degraded_total":112,"degraded_ratio":118.86607142857143,"recovering_objects_per_sec":210,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1682,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:10.093 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:10.093 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:10.400 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:10.402 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:10.416 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-15T03:19:09.017331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19371,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701212,"kb_used_data":1772,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786148,"statfs":{"total":386547056640,"available":382757015552,"internally_reserved":0,"allocated":1814528,"data_stored":875654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1682,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1682,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002445"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11990,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.847061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:01.847061+0000","last_peered":"2024-09-15T03:19:01.847061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:01.847061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14910,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840290+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:01.840290+0000","last_peered":"2024-09-15T03:19:01.840290+0000","last_clean":"2024-09-15T03:19:01.840290+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:01.840290+0000","last_undegraded":"2024-09-15T03:19:01.840290+0000","last_fullsized":"2024-09-15T03:19:01.840290+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.843273+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:01.843273+0000","last_peered":"2024-09-15T03:19:01.843273+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:01.843273+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15612,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839339+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:01.839339+0000","last_peered":"2024-09-15T03:19:01.839339+0000","last_clean":"2024-09-15T03:19:01.839339+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:01.839339+0000","last_undegraded":"2024-09-15T03:19:01.839339+0000","last_fullsized":"2024-09-15T03:19:01.839339+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T03:19:02.961706+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:02.961706+0000","last_peered":"2024-09-15T03:19:02.961706+0000","last_clean":"2024-09-15T03:19:02.961706+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:02.961706+0000","last_undegraded":"2024-09-15T03:19:02.961706+0000","last_fullsized":"2024-09-15T03:19:02.961706+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12096,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.845732+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:01.845732+0000","last_peered":"2024-09-15T03:19:01.845732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:01.845732+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.846508+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:01.846508+0000","last_peered":"2024-09-15T03:19:01.846508+0000","last_clean":"2024-09-15T03:19:01.846508+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:01.846508+0000","last_undegraded":"2024-09-15T03:19:01.846508+0000","last_fullsized":"2024-09-15T03:19:01.846508+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12223,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840293+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:01.840293+0000","last_peered":"2024-09-15T03:19:01.840293+0000","last_clean":"2024-09-15T03:19:01.840293+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:01.840293+0000","last_undegraded":"2024-09-15T03:19:01.840293+0000","last_fullsized":"2024-09-15T03:19:01.840293+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843855+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:01.843855+0000","last_peered":"2024-09-15T03:19:01.843855+0000","last_clean":"2024-09-15T03:19:01.843855+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:01.843855+0000","last_undegraded":"2024-09-15T03:19:01.843855+0000","last_fullsized":"2024-09-15T03:19:01.843855+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843859+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:01.843859+0000","last_peered":"2024-09-15T03:19:01.843859+0000","last_clean":"2024-09-15T03:19:01.843859+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:01.843859+0000","last_undegraded":"2024-09-15T03:19:01.843859+0000","last_fullsized":"2024-09-15T03:19:01.843859+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11988,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.844639+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:01.844639+0000","last_peered":"2024-09-15T03:19:01.844639+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:01.844639+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840343+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:01.840343+0000","last_peered":"2024-09-15T03:19:01.840343+0000","last_clean":"2024-09-15T03:19:01.840343+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:01.840343+0000","last_undegraded":"2024-09-15T03:19:01.840343+0000","last_fullsized":"2024-09-15T03:19:01.840343+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16320,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839511+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:01.839511+0000","last_peered":"2024-09-15T03:19:01.839511+0000","last_clean":"2024-09-15T03:19:01.839511+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:01.839511+0000","last_undegraded":"2024-09-15T03:19:01.839511+0000","last_fullsized":"2024-09-15T03:19:01.839511+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11794,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841036+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:01.841036+0000","last_peered":"2024-09-15T03:19:01.841036+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:01.841036+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12002,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841227+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:19:01.841227+0000","last_peered":"2024-09-15T03:19:01.841227+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:01.841227+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":13941,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:07.264158+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:07.264158+0000","last_peered":"2024-09-15T03:19:07.264158+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:07.264158+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":928,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1156}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19371,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932128,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439712,"statfs":{"total":96636764160,"available":95682265088,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919200,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452640,"statfs":{"total":96636764160,"available":95695503360,"internally_reserved":0,"allocated":409600,"data_stored":177241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.39600000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.53200000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930236,"kb_used_data":564,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441604,"statfs":{"total":96636764160,"available":95684202496,"internally_reserved":0,"allocated":577536,"data_stored":342031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.441},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.39800000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919648,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452192,"statfs":{"total":96636764160,"available":95695044608,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.627},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.441}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.40500000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:13.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:13.719 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:13.722 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:13.735 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-15T03:19:13.018462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20242,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692892,"kb_used_data":1772,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794468,"statfs":{"total":386547056640,"available":382765535232,"internally_reserved":0,"allocated":1814528,"data_stored":875654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1702,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.181461"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11990,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.847061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:01.847061+0000","last_peered":"2024-09-15T03:19:01.847061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:01.847061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14910,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840290+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:01.840290+0000","last_peered":"2024-09-15T03:19:01.840290+0000","last_clean":"2024-09-15T03:19:01.840290+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:01.840290+0000","last_undegraded":"2024-09-15T03:19:01.840290+0000","last_fullsized":"2024-09-15T03:19:01.840290+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.843273+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:01.843273+0000","last_peered":"2024-09-15T03:19:01.843273+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:01.843273+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15612,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839339+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:01.839339+0000","last_peered":"2024-09-15T03:19:01.839339+0000","last_clean":"2024-09-15T03:19:01.839339+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:01.839339+0000","last_undegraded":"2024-09-15T03:19:01.839339+0000","last_fullsized":"2024-09-15T03:19:01.839339+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T03:19:02.961706+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:02.961706+0000","last_peered":"2024-09-15T03:19:02.961706+0000","last_clean":"2024-09-15T03:19:02.961706+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:02.961706+0000","last_undegraded":"2024-09-15T03:19:02.961706+0000","last_fullsized":"2024-09-15T03:19:02.961706+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12096,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.845732+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:01.845732+0000","last_peered":"2024-09-15T03:19:01.845732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:01.845732+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.846508+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:01.846508+0000","last_peered":"2024-09-15T03:19:01.846508+0000","last_clean":"2024-09-15T03:19:01.846508+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:01.846508+0000","last_undegraded":"2024-09-15T03:19:01.846508+0000","last_fullsized":"2024-09-15T03:19:01.846508+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12223,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840293+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:01.840293+0000","last_peered":"2024-09-15T03:19:01.840293+0000","last_clean":"2024-09-15T03:19:01.840293+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:01.840293+0000","last_undegraded":"2024-09-15T03:19:01.840293+0000","last_fullsized":"2024-09-15T03:19:01.840293+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843855+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:01.843855+0000","last_peered":"2024-09-15T03:19:01.843855+0000","last_clean":"2024-09-15T03:19:01.843855+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:01.843855+0000","last_undegraded":"2024-09-15T03:19:01.843855+0000","last_fullsized":"2024-09-15T03:19:01.843855+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843859+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:01.843859+0000","last_peered":"2024-09-15T03:19:01.843859+0000","last_clean":"2024-09-15T03:19:01.843859+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:01.843859+0000","last_undegraded":"2024-09-15T03:19:01.843859+0000","last_fullsized":"2024-09-15T03:19:01.843859+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11988,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.844639+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:01.844639+0000","last_peered":"2024-09-15T03:19:01.844639+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:01.844639+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840343+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:01.840343+0000","last_peered":"2024-09-15T03:19:01.840343+0000","last_clean":"2024-09-15T03:19:01.840343+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:01.840343+0000","last_undegraded":"2024-09-15T03:19:01.840343+0000","last_fullsized":"2024-09-15T03:19:01.840343+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16320,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839511+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:01.839511+0000","last_peered":"2024-09-15T03:19:01.839511+0000","last_clean":"2024-09-15T03:19:01.839511+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:01.839511+0000","last_undegraded":"2024-09-15T03:19:01.839511+0000","last_fullsized":"2024-09-15T03:19:01.839511+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11794,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841036+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:01.841036+0000","last_peered":"2024-09-15T03:19:01.841036+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:01.841036+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12002,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841227+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:19:01.841227+0000","last_peered":"2024-09-15T03:19:01.841227+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:01.841227+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":15683,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:12.265479+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:12.265479+0000","last_peered":"2024-09-15T03:19:12.265479+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:12.265479+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1799,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20242,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932128,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439712,"statfs":{"total":96636764160,"available":95682265088,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919200,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452640,"statfs":{"total":96636764160,"available":95695503360,"internally_reserved":0,"allocated":409600,"data_stored":177241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.38100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921916,"kb_used_data":564,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449924,"statfs":{"total":96636764160,"available":95692722176,"internally_reserved":0,"allocated":577536,"data_stored":342031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.307},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.47499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.93799999999999994}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919648,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452192,"statfs":{"total":96636764160,"available":95695044608,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.70599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.61199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.80600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:13.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:14.038 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:14.040 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:14.056 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-15T03:19:13.018462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20242,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692892,"kb_used_data":1772,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794468,"statfs":{"total":386547056640,"available":382765535232,"internally_reserved":0,"allocated":1814528,"data_stored":875654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1702,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.181461"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11990,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.847061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:01.847061+0000","last_peered":"2024-09-15T03:19:01.847061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:01.847061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14910,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840290+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:01.840290+0000","last_peered":"2024-09-15T03:19:01.840290+0000","last_clean":"2024-09-15T03:19:01.840290+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:01.840290+0000","last_undegraded":"2024-09-15T03:19:01.840290+0000","last_fullsized":"2024-09-15T03:19:01.840290+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.843273+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:01.843273+0000","last_peered":"2024-09-15T03:19:01.843273+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:01.843273+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15612,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839339+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:01.839339+0000","last_peered":"2024-09-15T03:19:01.839339+0000","last_clean":"2024-09-15T03:19:01.839339+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:01.839339+0000","last_undegraded":"2024-09-15T03:19:01.839339+0000","last_fullsized":"2024-09-15T03:19:01.839339+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T03:19:02.961706+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:02.961706+0000","last_peered":"2024-09-15T03:19:02.961706+0000","last_clean":"2024-09-15T03:19:02.961706+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:02.961706+0000","last_undegraded":"2024-09-15T03:19:02.961706+0000","last_fullsized":"2024-09-15T03:19:02.961706+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12096,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.845732+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:01.845732+0000","last_peered":"2024-09-15T03:19:01.845732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:01.845732+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.846508+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:01.846508+0000","last_peered":"2024-09-15T03:19:01.846508+0000","last_clean":"2024-09-15T03:19:01.846508+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:01.846508+0000","last_undegraded":"2024-09-15T03:19:01.846508+0000","last_fullsized":"2024-09-15T03:19:01.846508+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12223,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840293+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:01.840293+0000","last_peered":"2024-09-15T03:19:01.840293+0000","last_clean":"2024-09-15T03:19:01.840293+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:01.840293+0000","last_undegraded":"2024-09-15T03:19:01.840293+0000","last_fullsized":"2024-09-15T03:19:01.840293+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843855+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:01.843855+0000","last_peered":"2024-09-15T03:19:01.843855+0000","last_clean":"2024-09-15T03:19:01.843855+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:01.843855+0000","last_undegraded":"2024-09-15T03:19:01.843855+0000","last_fullsized":"2024-09-15T03:19:01.843855+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843859+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:01.843859+0000","last_peered":"2024-09-15T03:19:01.843859+0000","last_clean":"2024-09-15T03:19:01.843859+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:01.843859+0000","last_undegraded":"2024-09-15T03:19:01.843859+0000","last_fullsized":"2024-09-15T03:19:01.843859+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11988,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.844639+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:01.844639+0000","last_peered":"2024-09-15T03:19:01.844639+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:01.844639+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840343+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:01.840343+0000","last_peered":"2024-09-15T03:19:01.840343+0000","last_clean":"2024-09-15T03:19:01.840343+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:01.840343+0000","last_undegraded":"2024-09-15T03:19:01.840343+0000","last_fullsized":"2024-09-15T03:19:01.840343+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16320,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839511+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:01.839511+0000","last_peered":"2024-09-15T03:19:01.839511+0000","last_clean":"2024-09-15T03:19:01.839511+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:01.839511+0000","last_undegraded":"2024-09-15T03:19:01.839511+0000","last_fullsized":"2024-09-15T03:19:01.839511+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11794,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841036+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:01.841036+0000","last_peered":"2024-09-15T03:19:01.841036+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:01.841036+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12002,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841227+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:19:01.841227+0000","last_peered":"2024-09-15T03:19:01.841227+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:01.841227+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":15683,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:12.265479+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:12.265479+0000","last_peered":"2024-09-15T03:19:12.265479+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:12.265479+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1799,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20242,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932128,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439712,"statfs":{"total":96636764160,"available":95682265088,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919200,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452640,"statfs":{"total":96636764160,"available":95695503360,"internally_reserved":0,"allocated":409600,"data_stored":177241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.38100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921916,"kb_used_data":564,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449924,"statfs":{"total":96636764160,"available":95692722176,"internally_reserved":0,"allocated":577536,"data_stored":342031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.307},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.47499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.93799999999999994}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919648,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452192,"statfs":{"total":96636764160,"available":95695044608,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.70599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.61199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.80600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:14.057 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:14.057 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11990, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.847061+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:01.847061+0000', 'last_peered': '2024-09-15T03:19:01.847061+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:01.847061+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12342, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.843273+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:19:01.843273+0000', 'last_peered': '2024-09-15T03:19:01.843273+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:01.843273+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12096, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.845732+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:01.845732+0000', 'last_peered': '2024-09-15T03:19:01.845732+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:01.845732+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11988, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.844639+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:19:01.844639+0000', 'last_peered': '2024-09-15T03:19:01.844639+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:01.844639+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11794, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.841036+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:01.841036+0000', 'last_peered': '2024-09-15T03:19:01.841036+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:01.841036+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:14.058 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:19:14.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 12002, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.841227+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:19:01.841227+0000', 'last_peered': '2024-09-15T03:19:01.841227+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:19:01.841227+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:14.059 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T03:19:14.059 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12007", 'reported_seq': 15683, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:12.265479+0000', 'last_change': '2024-09-15T03:19:01.812984+0000', 'last_active': '2024-09-15T03:19:12.265479+0000', 'last_peered': '2024-09-15T03:19:12.265479+0000', 'last_clean': '2024-09-15T03:05:04.607124+0000', 'last_became_active': '2024-09-15T03:06:09.306169+0000', 'last_became_peered': '2024-09-15T03:06:09.306169+0000', 'last_unstale': '2024-09-15T03:19:12.265479+0000', 'last_undegraded': '2024-09-15T03:06:08.918968+0000', 'last_fullsized': '2024-09-15T03:06:08.901311+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 48028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1799, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -285}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T03:19:14.059 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:14.443 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:14.444 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12442/112 objects degraded (11108.929%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":877,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":50,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3781521408,"bytes_avail":382765535232,"bytes_total":386547056640,"degraded_objects":12442,"degraded_total":112,"degraded_ratio":111.08928571428571,"recovering_objects_per_sec":167,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1702,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:14.462 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:14.462 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:14.767 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:14.771 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:14.783 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-15T03:19:14.349277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20242,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3692892,"kb_used_data":1772,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794468,"statfs":{"total":386547056640,"available":382765535232,"internally_reserved":0,"allocated":1814528,"data_stored":875654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1702,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1702,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.333641"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11990,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.847061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:01.847061+0000","last_peered":"2024-09-15T03:19:01.847061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:01.847061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14910,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840290+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:01.840290+0000","last_peered":"2024-09-15T03:19:01.840290+0000","last_clean":"2024-09-15T03:19:01.840290+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:01.840290+0000","last_undegraded":"2024-09-15T03:19:01.840290+0000","last_fullsized":"2024-09-15T03:19:01.840290+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.843273+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:01.843273+0000","last_peered":"2024-09-15T03:19:01.843273+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:01.843273+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15612,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839339+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:01.839339+0000","last_peered":"2024-09-15T03:19:01.839339+0000","last_clean":"2024-09-15T03:19:01.839339+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:01.839339+0000","last_undegraded":"2024-09-15T03:19:01.839339+0000","last_fullsized":"2024-09-15T03:19:01.839339+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T03:19:02.961706+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:02.961706+0000","last_peered":"2024-09-15T03:19:02.961706+0000","last_clean":"2024-09-15T03:19:02.961706+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:02.961706+0000","last_undegraded":"2024-09-15T03:19:02.961706+0000","last_fullsized":"2024-09-15T03:19:02.961706+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12096,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.845732+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:01.845732+0000","last_peered":"2024-09-15T03:19:01.845732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:01.845732+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.846508+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:01.846508+0000","last_peered":"2024-09-15T03:19:01.846508+0000","last_clean":"2024-09-15T03:19:01.846508+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:01.846508+0000","last_undegraded":"2024-09-15T03:19:01.846508+0000","last_fullsized":"2024-09-15T03:19:01.846508+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12223,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840293+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:01.840293+0000","last_peered":"2024-09-15T03:19:01.840293+0000","last_clean":"2024-09-15T03:19:01.840293+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:01.840293+0000","last_undegraded":"2024-09-15T03:19:01.840293+0000","last_fullsized":"2024-09-15T03:19:01.840293+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843855+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:01.843855+0000","last_peered":"2024-09-15T03:19:01.843855+0000","last_clean":"2024-09-15T03:19:01.843855+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:01.843855+0000","last_undegraded":"2024-09-15T03:19:01.843855+0000","last_fullsized":"2024-09-15T03:19:01.843855+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843859+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:01.843859+0000","last_peered":"2024-09-15T03:19:01.843859+0000","last_clean":"2024-09-15T03:19:01.843859+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:01.843859+0000","last_undegraded":"2024-09-15T03:19:01.843859+0000","last_fullsized":"2024-09-15T03:19:01.843859+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11988,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.844639+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:01.844639+0000","last_peered":"2024-09-15T03:19:01.844639+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:01.844639+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.840343+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:01.840343+0000","last_peered":"2024-09-15T03:19:01.840343+0000","last_clean":"2024-09-15T03:19:01.840343+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:01.840343+0000","last_undegraded":"2024-09-15T03:19:01.840343+0000","last_fullsized":"2024-09-15T03:19:01.840343+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16320,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.839511+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:01.839511+0000","last_peered":"2024-09-15T03:19:01.839511+0000","last_clean":"2024-09-15T03:19:01.839511+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:01.839511+0000","last_undegraded":"2024-09-15T03:19:01.839511+0000","last_fullsized":"2024-09-15T03:19:01.839511+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11794,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841036+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:01.841036+0000","last_peered":"2024-09-15T03:19:01.841036+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:01.841036+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12002,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841227+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:19:01.841227+0000","last_peered":"2024-09-15T03:19:01.841227+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:01.841227+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":15683,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:12.265479+0000","last_change":"2024-09-15T03:19:01.812984+0000","last_active":"2024-09-15T03:19:12.265479+0000","last_peered":"2024-09-15T03:19:12.265479+0000","last_clean":"2024-09-15T03:05:04.607124+0000","last_became_active":"2024-09-15T03:06:09.306169+0000","last_became_peered":"2024-09-15T03:06:09.306169+0000","last_unstale":"2024-09-15T03:19:12.265479+0000","last_undegraded":"2024-09-15T03:06:08.918968+0000","last_fullsized":"2024-09-15T03:06:08.901311+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1799,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-285}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20242,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932128,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439712,"statfs":{"total":96636764160,"available":95682265088,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919200,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452640,"statfs":{"total":96636764160,"available":95695503360,"internally_reserved":0,"allocated":409600,"data_stored":177241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.38100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921916,"kb_used_data":564,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449924,"statfs":{"total":96636764160,"available":95692722176,"internally_reserved":0,"allocated":577536,"data_stored":342031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.307},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.47499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.93799999999999994}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919648,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452192,"statfs":{"total":96636764160,"available":95695044608,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.70599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.61199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.80600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":167936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:17.785 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:18.097 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:18.099 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:18.112 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-15T03:19:17.019705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20529,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701228,"kb_used_data":1780,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786132,"statfs":{"total":386547056640,"available":382756999168,"internally_reserved":0,"allocated":1822720,"data_stored":874973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002373"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11990,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.847061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:01.847061+0000","last_peered":"2024-09-15T03:19:01.847061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:01.847061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14912,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.359158+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:14.359158+0000","last_peered":"2024-09-15T03:19:14.359158+0000","last_clean":"2024-09-15T03:19:14.359158+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:14.359158+0000","last_undegraded":"2024-09-15T03:19:14.359158+0000","last_fullsized":"2024-09-15T03:19:14.359158+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.843273+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:01.843273+0000","last_peered":"2024-09-15T03:19:01.843273+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:01.843273+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15614,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360256+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:14.360256+0000","last_peered":"2024-09-15T03:19:14.360256+0000","last_clean":"2024-09-15T03:19:14.360256+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:14.360256+0000","last_undegraded":"2024-09-15T03:19:14.360256+0000","last_fullsized":"2024-09-15T03:19:14.360256+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T03:19:02.961706+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:02.961706+0000","last_peered":"2024-09-15T03:19:02.961706+0000","last_clean":"2024-09-15T03:19:02.961706+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:02.961706+0000","last_undegraded":"2024-09-15T03:19:02.961706+0000","last_fullsized":"2024-09-15T03:19:02.961706+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12096,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.845732+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:01.845732+0000","last_peered":"2024-09-15T03:19:01.845732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:01.845732+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.846508+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:01.846508+0000","last_peered":"2024-09-15T03:19:01.846508+0000","last_clean":"2024-09-15T03:19:01.846508+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:01.846508+0000","last_undegraded":"2024-09-15T03:19:01.846508+0000","last_fullsized":"2024-09-15T03:19:01.846508+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358444+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:14.358444+0000","last_peered":"2024-09-15T03:19:14.358444+0000","last_clean":"2024-09-15T03:19:14.358444+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:14.358444+0000","last_undegraded":"2024-09-15T03:19:14.358444+0000","last_fullsized":"2024-09-15T03:19:14.358444+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843855+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:01.843855+0000","last_peered":"2024-09-15T03:19:01.843855+0000","last_clean":"2024-09-15T03:19:01.843855+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:01.843855+0000","last_undegraded":"2024-09-15T03:19:01.843855+0000","last_fullsized":"2024-09-15T03:19:01.843855+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843859+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:01.843859+0000","last_peered":"2024-09-15T03:19:01.843859+0000","last_clean":"2024-09-15T03:19:01.843859+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:01.843859+0000","last_undegraded":"2024-09-15T03:19:01.843859+0000","last_fullsized":"2024-09-15T03:19:01.843859+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11988,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.844639+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:01.844639+0000","last_peered":"2024-09-15T03:19:01.844639+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:01.844639+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15955,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358683+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:14.358683+0000","last_peered":"2024-09-15T03:19:14.358683+0000","last_clean":"2024-09-15T03:19:14.358683+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:14.358683+0000","last_undegraded":"2024-09-15T03:19:14.358683+0000","last_fullsized":"2024-09-15T03:19:14.358683+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16322,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360340+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:14.360340+0000","last_peered":"2024-09-15T03:19:14.360340+0000","last_clean":"2024-09-15T03:19:14.360340+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:14.360340+0000","last_undegraded":"2024-09-15T03:19:14.360340+0000","last_fullsized":"2024-09-15T03:19:14.360340+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11794,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841036+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:01.841036+0000","last_peered":"2024-09-15T03:19:01.841036+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:01.841036+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12002,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841227+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:19:01.841227+0000","last_peered":"2024-09-15T03:19:01.841227+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:01.841227+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12137,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.457715+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.457715+0000","last_peered":"2024-09-15T03:19:15.457715+0000","last_clean":"2024-09-15T03:19:15.457715+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.457715+0000","last_undegraded":"2024-09-15T03:19:15.457715+0000","last_fullsized":"2024-09-15T03:19:15.457715+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20529,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932128,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439712,"statfs":{"total":96636764160,"available":95682265088,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919200,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452640,"statfs":{"total":96636764160,"available":95695503360,"internally_reserved":0,"allocated":409600,"data_stored":177241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.38100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930252,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441588,"statfs":{"total":96636764160,"available":95684186112,"internally_reserved":0,"allocated":585728,"data_stored":341350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.33300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919648,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452192,"statfs":{"total":96636764160,"available":95695044608,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.70599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.61199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.80600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:18.114 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:18.414 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:18.417 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:18.428 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-15T03:19:17.019705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20529,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701228,"kb_used_data":1780,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786132,"statfs":{"total":386547056640,"available":382756999168,"internally_reserved":0,"allocated":1822720,"data_stored":874973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002373"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11990,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.847061+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:01.847061+0000","last_peered":"2024-09-15T03:19:01.847061+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:01.847061+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14912,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.359158+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:14.359158+0000","last_peered":"2024-09-15T03:19:14.359158+0000","last_clean":"2024-09-15T03:19:14.359158+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:14.359158+0000","last_undegraded":"2024-09-15T03:19:14.359158+0000","last_fullsized":"2024-09-15T03:19:14.359158+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.843273+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:01.843273+0000","last_peered":"2024-09-15T03:19:01.843273+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:01.843273+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15614,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360256+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:14.360256+0000","last_peered":"2024-09-15T03:19:14.360256+0000","last_clean":"2024-09-15T03:19:14.360256+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:14.360256+0000","last_undegraded":"2024-09-15T03:19:14.360256+0000","last_fullsized":"2024-09-15T03:19:14.360256+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16384,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T03:19:02.961706+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:02.961706+0000","last_peered":"2024-09-15T03:19:02.961706+0000","last_clean":"2024-09-15T03:19:02.961706+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:02.961706+0000","last_undegraded":"2024-09-15T03:19:02.961706+0000","last_fullsized":"2024-09-15T03:19:02.961706+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12096,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.845732+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:01.845732+0000","last_peered":"2024-09-15T03:19:01.845732+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:01.845732+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16160,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.846508+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:01.846508+0000","last_peered":"2024-09-15T03:19:01.846508+0000","last_clean":"2024-09-15T03:19:01.846508+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:01.846508+0000","last_undegraded":"2024-09-15T03:19:01.846508+0000","last_fullsized":"2024-09-15T03:19:01.846508+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358444+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:14.358444+0000","last_peered":"2024-09-15T03:19:14.358444+0000","last_clean":"2024-09-15T03:19:14.358444+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:14.358444+0000","last_undegraded":"2024-09-15T03:19:14.358444+0000","last_fullsized":"2024-09-15T03:19:14.358444+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843855+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:01.843855+0000","last_peered":"2024-09-15T03:19:01.843855+0000","last_clean":"2024-09-15T03:19:01.843855+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:01.843855+0000","last_undegraded":"2024-09-15T03:19:01.843855+0000","last_fullsized":"2024-09-15T03:19:01.843855+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-15T03:19:01.843859+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:01.843859+0000","last_peered":"2024-09-15T03:19:01.843859+0000","last_clean":"2024-09-15T03:19:01.843859+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:01.843859+0000","last_undegraded":"2024-09-15T03:19:01.843859+0000","last_fullsized":"2024-09-15T03:19:01.843859+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11988,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.844639+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:01.844639+0000","last_peered":"2024-09-15T03:19:01.844639+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:01.844639+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15955,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358683+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:14.358683+0000","last_peered":"2024-09-15T03:19:14.358683+0000","last_clean":"2024-09-15T03:19:14.358683+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:14.358683+0000","last_undegraded":"2024-09-15T03:19:14.358683+0000","last_fullsized":"2024-09-15T03:19:14.358683+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16322,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360340+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:14.360340+0000","last_peered":"2024-09-15T03:19:14.360340+0000","last_clean":"2024-09-15T03:19:14.360340+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:14.360340+0000","last_undegraded":"2024-09-15T03:19:14.360340+0000","last_fullsized":"2024-09-15T03:19:14.360340+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11794,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841036+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:01.841036+0000","last_peered":"2024-09-15T03:19:01.841036+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:01.841036+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12002,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:01.841227+0000","last_change":"2024-09-15T03:06:09.259440+0000","last_active":"2024-09-15T03:19:01.841227+0000","last_peered":"2024-09-15T03:19:01.841227+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:01.841227+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12137,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.457715+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.457715+0000","last_peered":"2024-09-15T03:19:15.457715+0000","last_clean":"2024-09-15T03:19:15.457715+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.457715+0000","last_undegraded":"2024-09-15T03:19:15.457715+0000","last_fullsized":"2024-09-15T03:19:15.457715+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20529,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932128,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439712,"statfs":{"total":96636764160,"available":95682265088,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919200,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452640,"statfs":{"total":96636764160,"available":95695503360,"internally_reserved":0,"allocated":409600,"data_stored":177241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.38100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.54600000000000004}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930252,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441588,"statfs":{"total":96636764160,"available":95684186112,"internally_reserved":0,"allocated":585728,"data_stored":341350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.33300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919648,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452192,"statfs":{"total":96636764160,"available":95695044608,"internally_reserved":0,"allocated":413696,"data_stored":178191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.70599999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.61199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.80600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:18.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:18.429 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11990, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.847061+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:01.847061+0000', 'last_peered': '2024-09-15T03:19:01.847061+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:01.847061+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:18.429 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12342, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.843273+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:19:01.843273+0000', 'last_peered': '2024-09-15T03:19:01.843273+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:01.843273+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12096, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.845732+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:01.845732+0000', 'last_peered': '2024-09-15T03:19:01.845732+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:01.845732+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11988, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.844639+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:19:01.844639+0000', 'last_peered': '2024-09-15T03:19:01.844639+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:01.844639+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11794, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.841036+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:01.841036+0000', 'last_peered': '2024-09-15T03:19:01.841036+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:01.841036+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:19:18.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 12002, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:01.841227+0000', 'last_change': '2024-09-15T03:06:09.259440+0000', 'last_active': '2024-09-15T03:19:01.841227+0000', 'last_peered': '2024-09-15T03:19:01.841227+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:19:01.841227+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1979, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:18.431 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:18.801 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:18.801 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12155/112 objects degraded (10852.679%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":881,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3790057472,"bytes_avail":382756999168,"bytes_total":386547056640,"degraded_objects":12155,"degraded_total":112,"degraded_ratio":108.52678571428571,"recovering_objects_per_sec":144,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1158,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:18.822 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:18.823 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:19.125 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:19.127 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:19.139 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-09-15T03:19:19.020364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21003,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701276,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786084,"statfs":{"total":386547056640,"available":382756950016,"internally_reserved":0,"allocated":1871872,"data_stored":897506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1632,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1632,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002389"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11992,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352742+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:14.352742+0000","last_peered":"2024-09-15T03:19:14.352742+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:14.352742+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14912,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.359158+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:14.359158+0000","last_peered":"2024-09-15T03:19:14.359158+0000","last_clean":"2024-09-15T03:19:14.359158+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:14.359158+0000","last_undegraded":"2024-09-15T03:19:14.359158+0000","last_fullsized":"2024-09-15T03:19:14.359158+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12344,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.357036+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:14.357036+0000","last_peered":"2024-09-15T03:19:14.357036+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:14.357036+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15614,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360256+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:14.360256+0000","last_peered":"2024-09-15T03:19:14.360256+0000","last_clean":"2024-09-15T03:19:14.360256+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:14.360256+0000","last_undegraded":"2024-09-15T03:19:14.360256+0000","last_fullsized":"2024-09-15T03:19:14.360256+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16386,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351736+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:14.351736+0000","last_peered":"2024-09-15T03:19:14.351736+0000","last_clean":"2024-09-15T03:19:14.351736+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:14.351736+0000","last_undegraded":"2024-09-15T03:19:14.351736+0000","last_fullsized":"2024-09-15T03:19:14.351736+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12098,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352394+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:14.352394+0000","last_peered":"2024-09-15T03:19:14.352394+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:14.352394+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.356880+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:14.356880+0000","last_peered":"2024-09-15T03:19:14.356880+0000","last_clean":"2024-09-15T03:19:14.356880+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:14.356880+0000","last_undegraded":"2024-09-15T03:19:14.356880+0000","last_fullsized":"2024-09-15T03:19:14.356880+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358444+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:14.358444+0000","last_peered":"2024-09-15T03:19:14.358444+0000","last_clean":"2024-09-15T03:19:14.358444+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:14.358444+0000","last_undegraded":"2024-09-15T03:19:14.358444+0000","last_fullsized":"2024-09-15T03:19:14.358444+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16146,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:14.351838+0000","last_peered":"2024-09-15T03:19:14.351838+0000","last_clean":"2024-09-15T03:19:14.351838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:14.351838+0000","last_undegraded":"2024-09-15T03:19:14.351838+0000","last_fullsized":"2024-09-15T03:19:14.351838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16202,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.357139+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:14.357139+0000","last_peered":"2024-09-15T03:19:14.357139+0000","last_clean":"2024-09-15T03:19:14.357139+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:14.357139+0000","last_undegraded":"2024-09-15T03:19:14.357139+0000","last_fullsized":"2024-09-15T03:19:14.357139+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11990,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.351992+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:14.351992+0000","last_peered":"2024-09-15T03:19:14.351992+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:14.351992+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15955,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358683+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:14.358683+0000","last_peered":"2024-09-15T03:19:14.358683+0000","last_clean":"2024-09-15T03:19:14.358683+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:14.358683+0000","last_undegraded":"2024-09-15T03:19:14.358683+0000","last_fullsized":"2024-09-15T03:19:14.358683+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16322,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360340+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:14.360340+0000","last_peered":"2024-09-15T03:19:14.360340+0000","last_clean":"2024-09-15T03:19:14.360340+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:14.360340+0000","last_undegraded":"2024-09-15T03:19:14.360340+0000","last_fullsized":"2024-09-15T03:19:14.360340+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11796,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352309+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:14.352309+0000","last_peered":"2024-09-15T03:19:14.352309+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:14.352309+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12953,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:17.220373+0000","last_change":"2024-09-15T03:19:13.857465+0000","last_active":"2024-09-15T03:19:17.220373+0000","last_peered":"2024-09-15T03:19:17.220373+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:17.220373+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12137,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.457715+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.457715+0000","last_peered":"2024-09-15T03:19:15.457715+0000","last_clean":"2024-09-15T03:19:15.457715+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.457715+0000","last_undegraded":"2024-09-15T03:19:15.457715+0000","last_fullsized":"2024-09-15T03:19:15.457715+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21003,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932144,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439696,"statfs":{"total":96636764160,"available":95682248704,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.78000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":425984,"data_stored":184752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.64200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.77000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.90500000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930252,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441588,"statfs":{"total":96636764160,"available":95684186112,"internally_reserved":0,"allocated":585728,"data_stored":341350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.33300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919664,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452176,"statfs":{"total":96636764160,"available":95695028224,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.435}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.45800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:22.141 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:22.464 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:22.467 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:22.480 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-15T03:19:21.021171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21003,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701276,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786084,"statfs":{"total":386547056640,"available":382756950016,"internally_reserved":0,"allocated":1871872,"data_stored":897506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002709"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11992,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352742+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:14.352742+0000","last_peered":"2024-09-15T03:19:14.352742+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:14.352742+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14912,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.359158+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:14.359158+0000","last_peered":"2024-09-15T03:19:14.359158+0000","last_clean":"2024-09-15T03:19:14.359158+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:14.359158+0000","last_undegraded":"2024-09-15T03:19:14.359158+0000","last_fullsized":"2024-09-15T03:19:14.359158+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12344,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.357036+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:14.357036+0000","last_peered":"2024-09-15T03:19:14.357036+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:14.357036+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15614,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360256+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:14.360256+0000","last_peered":"2024-09-15T03:19:14.360256+0000","last_clean":"2024-09-15T03:19:14.360256+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:14.360256+0000","last_undegraded":"2024-09-15T03:19:14.360256+0000","last_fullsized":"2024-09-15T03:19:14.360256+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16386,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351736+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:14.351736+0000","last_peered":"2024-09-15T03:19:14.351736+0000","last_clean":"2024-09-15T03:19:14.351736+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:14.351736+0000","last_undegraded":"2024-09-15T03:19:14.351736+0000","last_fullsized":"2024-09-15T03:19:14.351736+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12098,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352394+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:14.352394+0000","last_peered":"2024-09-15T03:19:14.352394+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:14.352394+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.356880+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:14.356880+0000","last_peered":"2024-09-15T03:19:14.356880+0000","last_clean":"2024-09-15T03:19:14.356880+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:14.356880+0000","last_undegraded":"2024-09-15T03:19:14.356880+0000","last_fullsized":"2024-09-15T03:19:14.356880+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358444+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:14.358444+0000","last_peered":"2024-09-15T03:19:14.358444+0000","last_clean":"2024-09-15T03:19:14.358444+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:14.358444+0000","last_undegraded":"2024-09-15T03:19:14.358444+0000","last_fullsized":"2024-09-15T03:19:14.358444+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16146,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:14.351838+0000","last_peered":"2024-09-15T03:19:14.351838+0000","last_clean":"2024-09-15T03:19:14.351838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:14.351838+0000","last_undegraded":"2024-09-15T03:19:14.351838+0000","last_fullsized":"2024-09-15T03:19:14.351838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16202,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.357139+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:14.357139+0000","last_peered":"2024-09-15T03:19:14.357139+0000","last_clean":"2024-09-15T03:19:14.357139+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:14.357139+0000","last_undegraded":"2024-09-15T03:19:14.357139+0000","last_fullsized":"2024-09-15T03:19:14.357139+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11990,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.351992+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:14.351992+0000","last_peered":"2024-09-15T03:19:14.351992+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:14.351992+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15955,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358683+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:14.358683+0000","last_peered":"2024-09-15T03:19:14.358683+0000","last_clean":"2024-09-15T03:19:14.358683+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:14.358683+0000","last_undegraded":"2024-09-15T03:19:14.358683+0000","last_fullsized":"2024-09-15T03:19:14.358683+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16322,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360340+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:14.360340+0000","last_peered":"2024-09-15T03:19:14.360340+0000","last_clean":"2024-09-15T03:19:14.360340+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:14.360340+0000","last_undegraded":"2024-09-15T03:19:14.360340+0000","last_fullsized":"2024-09-15T03:19:14.360340+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11796,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352309+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:14.352309+0000","last_peered":"2024-09-15T03:19:14.352309+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:14.352309+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12953,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:17.220373+0000","last_change":"2024-09-15T03:19:13.857465+0000","last_active":"2024-09-15T03:19:17.220373+0000","last_peered":"2024-09-15T03:19:17.220373+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:17.220373+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12137,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.457715+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.457715+0000","last_peered":"2024-09-15T03:19:15.457715+0000","last_clean":"2024-09-15T03:19:15.457715+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.457715+0000","last_undegraded":"2024-09-15T03:19:15.457715+0000","last_fullsized":"2024-09-15T03:19:15.457715+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21003,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932144,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439696,"statfs":{"total":96636764160,"available":95682248704,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.78000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":425984,"data_stored":184752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.64200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.77000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.90500000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930252,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441588,"statfs":{"total":96636764160,"available":95684186112,"internally_reserved":0,"allocated":585728,"data_stored":341350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.33300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919664,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452176,"statfs":{"total":96636764160,"available":95695028224,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.435}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.45800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:22.481 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:22.786 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:22.788 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:22.798 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-15T03:19:21.021171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21003,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701276,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373786084,"statfs":{"total":386547056640,"available":382756950016,"internally_reserved":0,"allocated":1871872,"data_stored":897506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-761,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002709"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11992,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352742+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:14.352742+0000","last_peered":"2024-09-15T03:19:14.352742+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:14.352742+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14912,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.359158+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:14.359158+0000","last_peered":"2024-09-15T03:19:14.359158+0000","last_clean":"2024-09-15T03:19:14.359158+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:14.359158+0000","last_undegraded":"2024-09-15T03:19:14.359158+0000","last_fullsized":"2024-09-15T03:19:14.359158+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12344,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.357036+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:14.357036+0000","last_peered":"2024-09-15T03:19:14.357036+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:14.357036+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15614,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360256+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:14.360256+0000","last_peered":"2024-09-15T03:19:14.360256+0000","last_clean":"2024-09-15T03:19:14.360256+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:14.360256+0000","last_undegraded":"2024-09-15T03:19:14.360256+0000","last_fullsized":"2024-09-15T03:19:14.360256+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16386,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351736+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:14.351736+0000","last_peered":"2024-09-15T03:19:14.351736+0000","last_clean":"2024-09-15T03:19:14.351736+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:14.351736+0000","last_undegraded":"2024-09-15T03:19:14.351736+0000","last_fullsized":"2024-09-15T03:19:14.351736+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12098,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352394+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:14.352394+0000","last_peered":"2024-09-15T03:19:14.352394+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:14.352394+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.356880+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:14.356880+0000","last_peered":"2024-09-15T03:19:14.356880+0000","last_clean":"2024-09-15T03:19:14.356880+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:14.356880+0000","last_undegraded":"2024-09-15T03:19:14.356880+0000","last_fullsized":"2024-09-15T03:19:14.356880+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358444+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:14.358444+0000","last_peered":"2024-09-15T03:19:14.358444+0000","last_clean":"2024-09-15T03:19:14.358444+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:14.358444+0000","last_undegraded":"2024-09-15T03:19:14.358444+0000","last_fullsized":"2024-09-15T03:19:14.358444+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16146,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:14.351838+0000","last_peered":"2024-09-15T03:19:14.351838+0000","last_clean":"2024-09-15T03:19:14.351838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:14.351838+0000","last_undegraded":"2024-09-15T03:19:14.351838+0000","last_fullsized":"2024-09-15T03:19:14.351838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16202,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.357139+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:14.357139+0000","last_peered":"2024-09-15T03:19:14.357139+0000","last_clean":"2024-09-15T03:19:14.357139+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:14.357139+0000","last_undegraded":"2024-09-15T03:19:14.357139+0000","last_fullsized":"2024-09-15T03:19:14.357139+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11990,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.351992+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:14.351992+0000","last_peered":"2024-09-15T03:19:14.351992+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:14.351992+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15955,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358683+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:14.358683+0000","last_peered":"2024-09-15T03:19:14.358683+0000","last_clean":"2024-09-15T03:19:14.358683+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:14.358683+0000","last_undegraded":"2024-09-15T03:19:14.358683+0000","last_fullsized":"2024-09-15T03:19:14.358683+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16322,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360340+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:14.360340+0000","last_peered":"2024-09-15T03:19:14.360340+0000","last_clean":"2024-09-15T03:19:14.360340+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:14.360340+0000","last_undegraded":"2024-09-15T03:19:14.360340+0000","last_fullsized":"2024-09-15T03:19:14.360340+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11796,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352309+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:14.352309+0000","last_peered":"2024-09-15T03:19:14.352309+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:14.352309+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12953,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:17.220373+0000","last_change":"2024-09-15T03:19:13.857465+0000","last_active":"2024-09-15T03:19:17.220373+0000","last_peered":"2024-09-15T03:19:17.220373+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:17.220373+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1505,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1503}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12137,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.457715+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.457715+0000","last_peered":"2024-09-15T03:19:15.457715+0000","last_clean":"2024-09-15T03:19:15.457715+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.457715+0000","last_undegraded":"2024-09-15T03:19:15.457715+0000","last_fullsized":"2024-09-15T03:19:15.457715+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21003,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932144,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439696,"statfs":{"total":96636764160,"available":95682248704,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.78000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":425984,"data_stored":184752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.64200000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.77000000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.90500000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930252,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441588,"statfs":{"total":96636764160,"available":95684186112,"internally_reserved":0,"allocated":585728,"data_stored":341350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":4.234,"15min":2.5019999999999998},"min":{"1min":0.20599999999999999,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":14.189,"5min":28.456,"15min":28.456},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.1579999999999999,"5min":2.4569999999999999,"15min":2},"min":{"1min":0.20699999999999999,"5min":0.19,"15min":0.187},"max":{"1min":11.987,"5min":49.399999999999999,"15min":49.399999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":2.9249999999999998,"15min":2.081},"min":{"1min":0.22600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":4.5090000000000003,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":4.0960000000000001,"15min":2.7570000000000001},"min":{"1min":0.22700000000000001,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":4.4699999999999998,"5min":28.145,"15min":28.145},"last":0.33300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:19 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":4.5069999999999997,"15min":2.641},"min":{"1min":0.25,"5min":0.25,"15min":0.222},"max":{"1min":7.0209999999999999,"5min":59.225999999999999,"15min":59.225999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.1619999999999999,"5min":2.371,"15min":1.883},"min":{"1min":0.255,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":21.401,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.40799999999999997}]}]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919664,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452176,"statfs":{"total":96636764160,"available":95695028224,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.435}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.45800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:22.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:22.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11992, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.352742+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:14.352742+0000', 'last_peered': '2024-09-15T03:19:14.352742+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:14.352742+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:22.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:22.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12344, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.357036+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:19:14.357036+0000', 'last_peered': '2024-09-15T03:19:14.357036+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:14.357036+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:22.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:22.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12098, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.352394+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:14.352394+0000', 'last_peered': '2024-09-15T03:19:14.352394+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:14.352394+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:22.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:22.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11990, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.351992+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:19:14.351992+0000', 'last_peered': '2024-09-15T03:19:14.351992+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:14.351992+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:22.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:22.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11796, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.352309+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:14.352309+0000', 'last_peered': '2024-09-15T03:19:14.352309+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:14.352309+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:22.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:19:22.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 12953, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:17.220373+0000', 'last_change': '2024-09-15T03:19:13.857465+0000', 'last_active': '2024-09-15T03:19:17.220373+0000', 'last_peered': '2024-09-15T03:19:17.220373+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:19:17.220373+0000', 'last_undegraded': '2024-09-15T03:06:08.886676+0000', 'last_fullsized': '2024-09-15T03:06:08.837180+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1505, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 487, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:22.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:23.165 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:23.166 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11681/112 objects degraded (10429.464%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":886,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3790106624,"bytes_avail":382756950016,"bytes_total":386547056640,"degraded_objects":11681,"degraded_total":112,"degraded_ratio":104.29464285714286,"recovering_objects_per_sec":95,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":761,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:23.179 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:23.179 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:23.474 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:23.477 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:23.488 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-09-15T03:19:23.021765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21849,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705428,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781932,"statfs":{"total":386547056640,"available":382752698368,"internally_reserved":0,"allocated":1871872,"data_stored":897506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1607,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.672488"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11992,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352742+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:14.352742+0000","last_peered":"2024-09-15T03:19:14.352742+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:14.352742+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14912,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.359158+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:14.359158+0000","last_peered":"2024-09-15T03:19:14.359158+0000","last_clean":"2024-09-15T03:19:14.359158+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:14.359158+0000","last_undegraded":"2024-09-15T03:19:14.359158+0000","last_fullsized":"2024-09-15T03:19:14.359158+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12344,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.357036+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:14.357036+0000","last_peered":"2024-09-15T03:19:14.357036+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:14.357036+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15614,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360256+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:14.360256+0000","last_peered":"2024-09-15T03:19:14.360256+0000","last_clean":"2024-09-15T03:19:14.360256+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:14.360256+0000","last_undegraded":"2024-09-15T03:19:14.360256+0000","last_fullsized":"2024-09-15T03:19:14.360256+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16386,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351736+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:14.351736+0000","last_peered":"2024-09-15T03:19:14.351736+0000","last_clean":"2024-09-15T03:19:14.351736+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:14.351736+0000","last_undegraded":"2024-09-15T03:19:14.351736+0000","last_fullsized":"2024-09-15T03:19:14.351736+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12098,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352394+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:14.352394+0000","last_peered":"2024-09-15T03:19:14.352394+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:14.352394+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.356880+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:14.356880+0000","last_peered":"2024-09-15T03:19:14.356880+0000","last_clean":"2024-09-15T03:19:14.356880+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:14.356880+0000","last_undegraded":"2024-09-15T03:19:14.356880+0000","last_fullsized":"2024-09-15T03:19:14.356880+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358444+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:14.358444+0000","last_peered":"2024-09-15T03:19:14.358444+0000","last_clean":"2024-09-15T03:19:14.358444+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:14.358444+0000","last_undegraded":"2024-09-15T03:19:14.358444+0000","last_fullsized":"2024-09-15T03:19:14.358444+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16146,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:14.351838+0000","last_peered":"2024-09-15T03:19:14.351838+0000","last_clean":"2024-09-15T03:19:14.351838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:14.351838+0000","last_undegraded":"2024-09-15T03:19:14.351838+0000","last_fullsized":"2024-09-15T03:19:14.351838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16202,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.357139+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:14.357139+0000","last_peered":"2024-09-15T03:19:14.357139+0000","last_clean":"2024-09-15T03:19:14.357139+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:14.357139+0000","last_undegraded":"2024-09-15T03:19:14.357139+0000","last_fullsized":"2024-09-15T03:19:14.357139+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11990,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.351992+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:14.351992+0000","last_peered":"2024-09-15T03:19:14.351992+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:14.351992+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15955,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358683+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:14.358683+0000","last_peered":"2024-09-15T03:19:14.358683+0000","last_clean":"2024-09-15T03:19:14.358683+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:14.358683+0000","last_undegraded":"2024-09-15T03:19:14.358683+0000","last_fullsized":"2024-09-15T03:19:14.358683+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16322,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360340+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:14.360340+0000","last_peered":"2024-09-15T03:19:14.360340+0000","last_clean":"2024-09-15T03:19:14.360340+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:14.360340+0000","last_undegraded":"2024-09-15T03:19:14.360340+0000","last_fullsized":"2024-09-15T03:19:14.360340+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11796,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352309+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:14.352309+0000","last_peered":"2024-09-15T03:19:14.352309+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:14.352309+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":14645,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:22.217950+0000","last_change":"2024-09-15T03:19:13.857465+0000","last_active":"2024-09-15T03:19:22.217950+0000","last_peered":"2024-09-15T03:19:22.217950+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:22.217950+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1333,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12137,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.457715+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.457715+0000","last_peered":"2024-09-15T03:19:15.457715+0000","last_clean":"2024-09-15T03:19:15.457715+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.457715+0000","last_undegraded":"2024-09-15T03:19:15.457715+0000","last_fullsized":"2024-09-15T03:19:15.457715+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21849,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932144,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439696,"statfs":{"total":96636764160,"available":95682248704,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.51600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.48299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":425984,"data_stored":184752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.44600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46899999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934404,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437436,"statfs":{"total":96636764160,"available":95679934464,"internally_reserved":0,"allocated":585728,"data_stored":341350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.40400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.54300000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.372},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.48199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919664,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452176,"statfs":{"total":96636764160,"available":95695028224,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.59899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:26.490 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:26.799 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:26.802 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:26.812 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-09-15T03:19:26.402374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21849,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705428,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781932,"statfs":{"total":386547056640,"available":382752698368,"internally_reserved":0,"allocated":1871872,"data_stored":897506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1607,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1607,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.043911"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11992,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352742+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:14.352742+0000","last_peered":"2024-09-15T03:19:14.352742+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:14.352742+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14912,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.359158+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:14.359158+0000","last_peered":"2024-09-15T03:19:14.359158+0000","last_clean":"2024-09-15T03:19:14.359158+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:14.359158+0000","last_undegraded":"2024-09-15T03:19:14.359158+0000","last_fullsized":"2024-09-15T03:19:14.359158+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12344,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.357036+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:14.357036+0000","last_peered":"2024-09-15T03:19:14.357036+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:14.357036+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15614,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360256+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:14.360256+0000","last_peered":"2024-09-15T03:19:14.360256+0000","last_clean":"2024-09-15T03:19:14.360256+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:14.360256+0000","last_undegraded":"2024-09-15T03:19:14.360256+0000","last_fullsized":"2024-09-15T03:19:14.360256+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16386,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351736+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:14.351736+0000","last_peered":"2024-09-15T03:19:14.351736+0000","last_clean":"2024-09-15T03:19:14.351736+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:14.351736+0000","last_undegraded":"2024-09-15T03:19:14.351736+0000","last_fullsized":"2024-09-15T03:19:14.351736+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12098,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352394+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:14.352394+0000","last_peered":"2024-09-15T03:19:14.352394+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:14.352394+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.356880+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:14.356880+0000","last_peered":"2024-09-15T03:19:14.356880+0000","last_clean":"2024-09-15T03:19:14.356880+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:14.356880+0000","last_undegraded":"2024-09-15T03:19:14.356880+0000","last_fullsized":"2024-09-15T03:19:14.356880+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12225,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358444+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:14.358444+0000","last_peered":"2024-09-15T03:19:14.358444+0000","last_clean":"2024-09-15T03:19:14.358444+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:14.358444+0000","last_undegraded":"2024-09-15T03:19:14.358444+0000","last_fullsized":"2024-09-15T03:19:14.358444+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16146,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:14.351838+0000","last_peered":"2024-09-15T03:19:14.351838+0000","last_clean":"2024-09-15T03:19:14.351838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:14.351838+0000","last_undegraded":"2024-09-15T03:19:14.351838+0000","last_fullsized":"2024-09-15T03:19:14.351838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16202,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.357139+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:14.357139+0000","last_peered":"2024-09-15T03:19:14.357139+0000","last_clean":"2024-09-15T03:19:14.357139+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:14.357139+0000","last_undegraded":"2024-09-15T03:19:14.357139+0000","last_fullsized":"2024-09-15T03:19:14.357139+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11990,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.351992+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:14.351992+0000","last_peered":"2024-09-15T03:19:14.351992+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:14.351992+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15955,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.358683+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:14.358683+0000","last_peered":"2024-09-15T03:19:14.358683+0000","last_clean":"2024-09-15T03:19:14.358683+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:14.358683+0000","last_undegraded":"2024-09-15T03:19:14.358683+0000","last_fullsized":"2024-09-15T03:19:14.358683+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16322,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.360340+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:14.360340+0000","last_peered":"2024-09-15T03:19:14.360340+0000","last_clean":"2024-09-15T03:19:14.360340+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:14.360340+0000","last_undegraded":"2024-09-15T03:19:14.360340+0000","last_fullsized":"2024-09-15T03:19:14.360340+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11796,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352309+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:14.352309+0000","last_peered":"2024-09-15T03:19:14.352309+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:14.352309+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":14645,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:22.217950+0000","last_change":"2024-09-15T03:19:13.857465+0000","last_active":"2024-09-15T03:19:22.217950+0000","last_peered":"2024-09-15T03:19:22.217950+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:22.217950+0000","last_undegraded":"2024-09-15T03:06:08.886676+0000","last_fullsized":"2024-09-15T03:06:08.837180+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":659,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1333,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12137,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.457715+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.457715+0000","last_peered":"2024-09-15T03:19:15.457715+0000","last_clean":"2024-09-15T03:19:15.457715+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.457715+0000","last_undegraded":"2024-09-15T03:19:15.457715+0000","last_fullsized":"2024-09-15T03:19:15.457715+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21849,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":172032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932144,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439696,"statfs":{"total":96636764160,"available":95682248704,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.51600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.48299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":425984,"data_stored":184752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.44600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46899999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934404,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437436,"statfs":{"total":96636764160,"available":95679934464,"internally_reserved":0,"allocated":585728,"data_stored":341350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.40400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.54300000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.372},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.48199999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919664,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452176,"statfs":{"total":96636764160,"available":95695028224,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.59899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:26.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:27.129 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:27.132 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:27.143 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-15T03:19:27.023159+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20529,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697096,"kb_used_data":1816,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790264,"statfs":{"total":386547056640,"available":382761230336,"internally_reserved":0,"allocated":1859584,"data_stored":885218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003453"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11992,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352742+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:14.352742+0000","last_peered":"2024-09-15T03:19:14.352742+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:14.352742+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14913,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364839+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:15.364839+0000","last_peered":"2024-09-15T03:19:15.364839+0000","last_clean":"2024-09-15T03:19:15.364839+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:15.364839+0000","last_undegraded":"2024-09-15T03:19:15.364839+0000","last_fullsized":"2024-09-15T03:19:15.364839+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12344,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.357036+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:14.357036+0000","last_peered":"2024-09-15T03:19:14.357036+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:14.357036+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15615,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364298+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:15.364298+0000","last_peered":"2024-09-15T03:19:15.364298+0000","last_clean":"2024-09-15T03:19:15.364298+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:15.364298+0000","last_undegraded":"2024-09-15T03:19:15.364298+0000","last_fullsized":"2024-09-15T03:19:15.364298+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16386,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351736+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:14.351736+0000","last_peered":"2024-09-15T03:19:14.351736+0000","last_clean":"2024-09-15T03:19:14.351736+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:14.351736+0000","last_undegraded":"2024-09-15T03:19:14.351736+0000","last_fullsized":"2024-09-15T03:19:14.351736+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12098,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352394+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:14.352394+0000","last_peered":"2024-09-15T03:19:14.352394+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:14.352394+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.356880+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:14.356880+0000","last_peered":"2024-09-15T03:19:14.356880+0000","last_clean":"2024-09-15T03:19:14.356880+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:14.356880+0000","last_undegraded":"2024-09-15T03:19:14.356880+0000","last_fullsized":"2024-09-15T03:19:14.356880+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364479+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:15.364479+0000","last_peered":"2024-09-15T03:19:15.364479+0000","last_clean":"2024-09-15T03:19:15.364479+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:15.364479+0000","last_undegraded":"2024-09-15T03:19:15.364479+0000","last_fullsized":"2024-09-15T03:19:15.364479+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16146,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.351838+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:14.351838+0000","last_peered":"2024-09-15T03:19:14.351838+0000","last_clean":"2024-09-15T03:19:14.351838+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:14.351838+0000","last_undegraded":"2024-09-15T03:19:14.351838+0000","last_fullsized":"2024-09-15T03:19:14.351838+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16202,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T03:19:14.357139+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:14.357139+0000","last_peered":"2024-09-15T03:19:14.357139+0000","last_clean":"2024-09-15T03:19:14.357139+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:14.357139+0000","last_undegraded":"2024-09-15T03:19:14.357139+0000","last_fullsized":"2024-09-15T03:19:14.357139+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":11990,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.351992+0000","last_change":"2024-09-15T03:06:09.261701+0000","last_active":"2024-09-15T03:19:14.351992+0000","last_peered":"2024-09-15T03:19:14.351992+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:14.351992+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2059}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15956,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364780+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:15.364780+0000","last_peered":"2024-09-15T03:19:15.364780+0000","last_clean":"2024-09-15T03:19:15.364780+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:15.364780+0000","last_undegraded":"2024-09-15T03:19:15.364780+0000","last_fullsized":"2024-09-15T03:19:15.364780+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16323,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364612+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:15.364612+0000","last_peered":"2024-09-15T03:19:15.364612+0000","last_clean":"2024-09-15T03:19:15.364612+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:15.364612+0000","last_undegraded":"2024-09-15T03:19:15.364612+0000","last_fullsized":"2024-09-15T03:19:15.364612+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11796,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:14.352309+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:14.352309+0000","last_peered":"2024-09-15T03:19:14.352309+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:14.352309+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11995,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-15T03:19:26.411040+0000","last_change":"2024-09-15T03:19:26.410892+0000","last_active":"2024-09-15T03:19:26.410720+0000","last_peered":"2024-09-15T03:17:42.479969+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:26.411040+0000","last_undegraded":"2024-09-15T03:19:26.411040+0000","last_fullsized":"2024-09-15T03:19:26.411040+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12138,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.458539+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.458539+0000","last_peered":"2024-09-15T03:19:15.458539+0000","last_clean":"2024-09-15T03:19:15.458539+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.458539+0000","last_undegraded":"2024-09-15T03:19:15.458539+0000","last_fullsized":"2024-09-15T03:19:15.458539+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20529,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932144,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439696,"statfs":{"total":96636764160,"available":95682248704,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.51600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.58399999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.48299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919216,"kb_used_data":416,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452624,"statfs":{"total":96636764160,"available":95695486976,"internally_reserved":0,"allocated":425984,"data_stored":184752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.44600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.46899999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926072,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445768,"statfs":{"total":96636764160,"available":95688466432,"internally_reserved":0,"allocated":573440,"data_stored":329062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.747}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.59699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919664,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452176,"statfs":{"total":96636764160,"available":95695028224,"internally_reserved":0,"allocated":430080,"data_stored":185702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.63100000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.59899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.54600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:27.144 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11992, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.352742+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:14.352742+0000', 'last_peered': '2024-09-15T03:19:14.352742+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:14.352742+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12344, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.357036+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:19:14.357036+0000', 'last_peered': '2024-09-15T03:19:14.357036+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:14.357036+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12098, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.352394+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:14.352394+0000', 'last_peered': '2024-09-15T03:19:14.352394+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:14.352394+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 11990, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.351992+0000', 'last_change': '2024-09-15T03:06:09.261701+0000', 'last_active': '2024-09-15T03:19:14.351992+0000', 'last_peered': '2024-09-15T03:19:14.351992+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:14.351992+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2065, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2059}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11796, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:14.352309+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:14.352309+0000', 'last_peered': '2024-09-15T03:19:14.352309+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:14.352309+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:27.145 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:19:27.146 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11995, 'reported_epoch': 52, 'state': 'peering', 'last_fresh': '2024-09-15T03:19:26.411040+0000', 'last_change': '2024-09-15T03:19:26.410892+0000', 'last_active': '2024-09-15T03:19:26.410720+0000', 'last_peered': '2024-09-15T03:17:42.479969+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:19:26.411040+0000', 'last_undegraded': '2024-09-15T03:19:26.411040+0000', 'last_fullsized': '2024-09-15T03:19:26.411040+0000', 'mapping_epoch': 52, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [1, 2, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-15T03:19:27.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:27.554 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:27.555 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg peering","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10176/112 objects degraded (9085.714%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":890,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"peering","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3785826304,"bytes_avail":382761230336,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":10176,"degraded_total":112,"degraded_ratio":90.857142857142861},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:27.572 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T03:19:27.573 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:27.908 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:27.910 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:27.921 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-15T03:19:27.424518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20730,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697116,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790244,"statfs":{"total":386547056640,"available":382761209856,"internally_reserved":0,"allocated":1884160,"data_stored":896405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-273,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.404153"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11993,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.360145+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:15.360145+0000","last_peered":"2024-09-15T03:19:15.360145+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:15.360145+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14913,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364839+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:15.364839+0000","last_peered":"2024-09-15T03:19:15.364839+0000","last_clean":"2024-09-15T03:19:15.364839+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:15.364839+0000","last_undegraded":"2024-09-15T03:19:15.364839+0000","last_fullsized":"2024-09-15T03:19:15.364839+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12345,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.365939+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:15.365939+0000","last_peered":"2024-09-15T03:19:15.365939+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:15.365939+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15615,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364298+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:15.364298+0000","last_peered":"2024-09-15T03:19:15.364298+0000","last_clean":"2024-09-15T03:19:15.364298+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:15.364298+0000","last_undegraded":"2024-09-15T03:19:15.364298+0000","last_fullsized":"2024-09-15T03:19:15.364298+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16387,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.359546+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:15.359546+0000","last_peered":"2024-09-15T03:19:15.359546+0000","last_clean":"2024-09-15T03:19:15.359546+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:15.359546+0000","last_undegraded":"2024-09-15T03:19:15.359546+0000","last_fullsized":"2024-09-15T03:19:15.359546+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12099,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.360132+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:15.360132+0000","last_peered":"2024-09-15T03:19:15.360132+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:15.360132+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16163,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.367148+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:15.367148+0000","last_peered":"2024-09-15T03:19:15.367148+0000","last_clean":"2024-09-15T03:19:15.367148+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:15.367148+0000","last_undegraded":"2024-09-15T03:19:15.367148+0000","last_fullsized":"2024-09-15T03:19:15.367148+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364479+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:15.364479+0000","last_peered":"2024-09-15T03:19:15.364479+0000","last_clean":"2024-09-15T03:19:15.364479+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:15.364479+0000","last_undegraded":"2024-09-15T03:19:15.364479+0000","last_fullsized":"2024-09-15T03:19:15.364479+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16147,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.359710+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:15.359710+0000","last_peered":"2024-09-15T03:19:15.359710+0000","last_clean":"2024-09-15T03:19:15.359710+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:15.359710+0000","last_undegraded":"2024-09-15T03:19:15.359710+0000","last_fullsized":"2024-09-15T03:19:15.359710+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16203,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.365705+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:15.365705+0000","last_peered":"2024-09-15T03:19:15.365705+0000","last_clean":"2024-09-15T03:19:15.365705+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:15.365705+0000","last_undegraded":"2024-09-15T03:19:15.365705+0000","last_fullsized":"2024-09-15T03:19:15.365705+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":12394,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.392286+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:27.392286+0000","last_peered":"2024-09-15T03:19:27.392286+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:27.392286+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15956,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364780+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:15.364780+0000","last_peered":"2024-09-15T03:19:15.364780+0000","last_clean":"2024-09-15T03:19:15.364780+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:15.364780+0000","last_undegraded":"2024-09-15T03:19:15.364780+0000","last_fullsized":"2024-09-15T03:19:15.364780+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16323,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364612+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:15.364612+0000","last_peered":"2024-09-15T03:19:15.364612+0000","last_clean":"2024-09-15T03:19:15.364612+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:15.364612+0000","last_undegraded":"2024-09-15T03:19:15.364612+0000","last_fullsized":"2024-09-15T03:19:15.364612+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.359958+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:15.359958+0000","last_peered":"2024-09-15T03:19:15.359958+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:15.359958+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11995,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-15T03:19:26.411040+0000","last_change":"2024-09-15T03:19:26.410892+0000","last_active":"2024-09-15T03:19:26.410720+0000","last_peered":"2024-09-15T03:17:42.479969+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:26.411040+0000","last_undegraded":"2024-09-15T03:19:26.411040+0000","last_fullsized":"2024-09-15T03:19:26.411040+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12138,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.458539+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.458539+0000","last_peered":"2024-09-15T03:19:15.458539+0000","last_clean":"2024-09-15T03:19:15.458539+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.458539+0000","last_undegraded":"2024-09-15T03:19:15.458539+0000","last_fullsized":"2024-09-15T03:19:15.458539+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20730,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932152,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439688,"statfs":{"total":96636764160,"available":95682240512,"internally_reserved":0,"allocated":438272,"data_stored":189431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.505}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.44400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919220,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452620,"statfs":{"total":96636764160,"available":95695482880,"internally_reserved":0,"allocated":434176,"data_stored":188481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.373}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.54900000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.42099999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926072,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445768,"statfs":{"total":96636764160,"available":95688466432,"internally_reserved":0,"allocated":573440,"data_stored":329062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.747}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.59699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919672,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452168,"statfs":{"total":96636764160,"available":95695020032,"internally_reserved":0,"allocated":438272,"data_stored":189431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.55400000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.31},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.47299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.503},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:30.923 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:31.243 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:31.245 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:31.256 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-15T03:19:31.024276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20730,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697116,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790244,"statfs":{"total":386547056640,"available":382761209856,"internally_reserved":0,"allocated":1884160,"data_stored":896405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002511"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11993,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.360145+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:15.360145+0000","last_peered":"2024-09-15T03:19:15.360145+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:15.360145+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14913,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364839+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:15.364839+0000","last_peered":"2024-09-15T03:19:15.364839+0000","last_clean":"2024-09-15T03:19:15.364839+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:15.364839+0000","last_undegraded":"2024-09-15T03:19:15.364839+0000","last_fullsized":"2024-09-15T03:19:15.364839+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12345,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.365939+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:15.365939+0000","last_peered":"2024-09-15T03:19:15.365939+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:15.365939+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15615,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364298+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:15.364298+0000","last_peered":"2024-09-15T03:19:15.364298+0000","last_clean":"2024-09-15T03:19:15.364298+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:15.364298+0000","last_undegraded":"2024-09-15T03:19:15.364298+0000","last_fullsized":"2024-09-15T03:19:15.364298+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16387,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.359546+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:15.359546+0000","last_peered":"2024-09-15T03:19:15.359546+0000","last_clean":"2024-09-15T03:19:15.359546+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:15.359546+0000","last_undegraded":"2024-09-15T03:19:15.359546+0000","last_fullsized":"2024-09-15T03:19:15.359546+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12099,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.360132+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:15.360132+0000","last_peered":"2024-09-15T03:19:15.360132+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:15.360132+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16163,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.367148+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:15.367148+0000","last_peered":"2024-09-15T03:19:15.367148+0000","last_clean":"2024-09-15T03:19:15.367148+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:15.367148+0000","last_undegraded":"2024-09-15T03:19:15.367148+0000","last_fullsized":"2024-09-15T03:19:15.367148+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364479+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:15.364479+0000","last_peered":"2024-09-15T03:19:15.364479+0000","last_clean":"2024-09-15T03:19:15.364479+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:15.364479+0000","last_undegraded":"2024-09-15T03:19:15.364479+0000","last_fullsized":"2024-09-15T03:19:15.364479+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16147,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.359710+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:15.359710+0000","last_peered":"2024-09-15T03:19:15.359710+0000","last_clean":"2024-09-15T03:19:15.359710+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:15.359710+0000","last_undegraded":"2024-09-15T03:19:15.359710+0000","last_fullsized":"2024-09-15T03:19:15.359710+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16203,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.365705+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:15.365705+0000","last_peered":"2024-09-15T03:19:15.365705+0000","last_clean":"2024-09-15T03:19:15.365705+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:15.365705+0000","last_undegraded":"2024-09-15T03:19:15.365705+0000","last_fullsized":"2024-09-15T03:19:15.365705+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":12394,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.392286+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:27.392286+0000","last_peered":"2024-09-15T03:19:27.392286+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:27.392286+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15956,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364780+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:15.364780+0000","last_peered":"2024-09-15T03:19:15.364780+0000","last_clean":"2024-09-15T03:19:15.364780+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:15.364780+0000","last_undegraded":"2024-09-15T03:19:15.364780+0000","last_fullsized":"2024-09-15T03:19:15.364780+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16323,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364612+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:15.364612+0000","last_peered":"2024-09-15T03:19:15.364612+0000","last_clean":"2024-09-15T03:19:15.364612+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:15.364612+0000","last_undegraded":"2024-09-15T03:19:15.364612+0000","last_fullsized":"2024-09-15T03:19:15.364612+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.359958+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:15.359958+0000","last_peered":"2024-09-15T03:19:15.359958+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:15.359958+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11995,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-15T03:19:26.411040+0000","last_change":"2024-09-15T03:19:26.410892+0000","last_active":"2024-09-15T03:19:26.410720+0000","last_peered":"2024-09-15T03:17:42.479969+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:26.411040+0000","last_undegraded":"2024-09-15T03:19:26.411040+0000","last_fullsized":"2024-09-15T03:19:26.411040+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12138,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.458539+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.458539+0000","last_peered":"2024-09-15T03:19:15.458539+0000","last_clean":"2024-09-15T03:19:15.458539+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.458539+0000","last_undegraded":"2024-09-15T03:19:15.458539+0000","last_fullsized":"2024-09-15T03:19:15.458539+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20730,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932152,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439688,"statfs":{"total":96636764160,"available":95682240512,"internally_reserved":0,"allocated":438272,"data_stored":189431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.505}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.44400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919220,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452620,"statfs":{"total":96636764160,"available":95695482880,"internally_reserved":0,"allocated":434176,"data_stored":188481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.373}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.54900000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.42099999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926072,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445768,"statfs":{"total":96636764160,"available":95688466432,"internally_reserved":0,"allocated":573440,"data_stored":329062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.747}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.59699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919672,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452168,"statfs":{"total":96636764160,"available":95695020032,"internally_reserved":0,"allocated":438272,"data_stored":189431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.55400000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.31},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.47299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.503},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:31.257 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:31.559 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:31.562 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:31.573 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-15T03:19:31.024276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20730,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697116,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790244,"statfs":{"total":386547056640,"available":382761209856,"internally_reserved":0,"allocated":1884160,"data_stored":896405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002511"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11993,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.360145+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:15.360145+0000","last_peered":"2024-09-15T03:19:15.360145+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:15.360145+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14913,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364839+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:15.364839+0000","last_peered":"2024-09-15T03:19:15.364839+0000","last_clean":"2024-09-15T03:19:15.364839+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:15.364839+0000","last_undegraded":"2024-09-15T03:19:15.364839+0000","last_fullsized":"2024-09-15T03:19:15.364839+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12345,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.365939+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:15.365939+0000","last_peered":"2024-09-15T03:19:15.365939+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:15.365939+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15615,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364298+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:15.364298+0000","last_peered":"2024-09-15T03:19:15.364298+0000","last_clean":"2024-09-15T03:19:15.364298+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:15.364298+0000","last_undegraded":"2024-09-15T03:19:15.364298+0000","last_fullsized":"2024-09-15T03:19:15.364298+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16387,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.359546+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:15.359546+0000","last_peered":"2024-09-15T03:19:15.359546+0000","last_clean":"2024-09-15T03:19:15.359546+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:15.359546+0000","last_undegraded":"2024-09-15T03:19:15.359546+0000","last_fullsized":"2024-09-15T03:19:15.359546+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12099,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.360132+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:15.360132+0000","last_peered":"2024-09-15T03:19:15.360132+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:15.360132+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16163,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.367148+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:15.367148+0000","last_peered":"2024-09-15T03:19:15.367148+0000","last_clean":"2024-09-15T03:19:15.367148+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:15.367148+0000","last_undegraded":"2024-09-15T03:19:15.367148+0000","last_fullsized":"2024-09-15T03:19:15.367148+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364479+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:15.364479+0000","last_peered":"2024-09-15T03:19:15.364479+0000","last_clean":"2024-09-15T03:19:15.364479+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:15.364479+0000","last_undegraded":"2024-09-15T03:19:15.364479+0000","last_fullsized":"2024-09-15T03:19:15.364479+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16147,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.359710+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:15.359710+0000","last_peered":"2024-09-15T03:19:15.359710+0000","last_clean":"2024-09-15T03:19:15.359710+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:15.359710+0000","last_undegraded":"2024-09-15T03:19:15.359710+0000","last_fullsized":"2024-09-15T03:19:15.359710+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16203,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.365705+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:15.365705+0000","last_peered":"2024-09-15T03:19:15.365705+0000","last_clean":"2024-09-15T03:19:15.365705+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:15.365705+0000","last_undegraded":"2024-09-15T03:19:15.365705+0000","last_fullsized":"2024-09-15T03:19:15.365705+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":12394,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.392286+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:27.392286+0000","last_peered":"2024-09-15T03:19:27.392286+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:27.392286+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15956,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364780+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:15.364780+0000","last_peered":"2024-09-15T03:19:15.364780+0000","last_clean":"2024-09-15T03:19:15.364780+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:15.364780+0000","last_undegraded":"2024-09-15T03:19:15.364780+0000","last_fullsized":"2024-09-15T03:19:15.364780+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16323,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364612+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:15.364612+0000","last_peered":"2024-09-15T03:19:15.364612+0000","last_clean":"2024-09-15T03:19:15.364612+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:15.364612+0000","last_undegraded":"2024-09-15T03:19:15.364612+0000","last_fullsized":"2024-09-15T03:19:15.364612+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.359958+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:15.359958+0000","last_peered":"2024-09-15T03:19:15.359958+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:15.359958+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11995,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-15T03:19:26.411040+0000","last_change":"2024-09-15T03:19:26.410892+0000","last_active":"2024-09-15T03:19:26.410720+0000","last_peered":"2024-09-15T03:17:42.479969+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:26.411040+0000","last_undegraded":"2024-09-15T03:19:26.411040+0000","last_fullsized":"2024-09-15T03:19:26.411040+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12138,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.458539+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.458539+0000","last_peered":"2024-09-15T03:19:15.458539+0000","last_clean":"2024-09-15T03:19:15.458539+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.458539+0000","last_undegraded":"2024-09-15T03:19:15.458539+0000","last_fullsized":"2024-09-15T03:19:15.458539+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20730,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932152,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439688,"statfs":{"total":96636764160,"available":95682240512,"internally_reserved":0,"allocated":438272,"data_stored":189431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.505}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.44400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919220,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452620,"statfs":{"total":96636764160,"available":95695482880,"internally_reserved":0,"allocated":434176,"data_stored":188481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.373}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.54900000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.42099999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926072,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445768,"statfs":{"total":96636764160,"available":95688466432,"internally_reserved":0,"allocated":573440,"data_stored":329062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.747}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.59699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919672,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452168,"statfs":{"total":96636764160,"available":95695020032,"internally_reserved":0,"allocated":438272,"data_stored":189431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.55400000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.31},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.47299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.503},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11993, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:15.360145+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:15.360145+0000', 'last_peered': '2024-09-15T03:19:15.360145+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:15.360145+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12345, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:15.365939+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:19:15.365939+0000', 'last_peered': '2024-09-15T03:19:15.365939+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:15.365939+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12099, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:15.360132+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:15.360132+0000', 'last_peered': '2024-09-15T03:19:15.360132+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:15.360132+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 12394, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:27.392286+0000', 'last_change': '2024-09-15T03:19:25.848067+0000', 'last_active': '2024-09-15T03:19:27.392286+0000', 'last_peered': '2024-09-15T03:19:27.392286+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:27.392286+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1864, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 220, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1861}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:31.575 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11797, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:15.359958+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:15.359958+0000', 'last_peered': '2024-09-15T03:19:15.359958+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:15.359958+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:31.576 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T03:19:31.576 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11881", 'reported_seq': 11995, 'reported_epoch': 52, 'state': 'peering', 'last_fresh': '2024-09-15T03:19:26.411040+0000', 'last_change': '2024-09-15T03:19:26.410892+0000', 'last_active': '2024-09-15T03:19:26.410720+0000', 'last_peered': '2024-09-15T03:17:42.479969+0000', 'last_clean': '2024-09-15T03:05:05.401750+0000', 'last_became_active': '2024-09-15T03:06:09.251548+0000', 'last_became_peered': '2024-09-15T03:06:09.251548+0000', 'last_unstale': '2024-09-15T03:19:26.411040+0000', 'last_undegraded': '2024-09-15T03:19:26.411040+0000', 'last_fullsized': '2024-09-15T03:19:26.411040+0000', 'mapping_epoch': 52, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11879, 'num_write_kb': 47525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [1, 2, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-15T03:19:31.576 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:31.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:19:31.874+0000 7f0abf7fe700 0 --2- 172.21.15.60:0/1776139595 >> v2:172.21.15.60:3300/0 conn(0x7f0ac01238f0 0x7f0ac0123cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:19:32.168 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:32.168 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg peering","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9975/112 objects degraded (8906.250%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":895,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3785846784,"bytes_avail":382761209856,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":9975,"degraded_total":112,"degraded_ratio":89.0625},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:32.181 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T03:19:32.182 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:32.488 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:32.492 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:32.505 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-15T03:19:31.024276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20730,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697116,"kb_used_data":1840,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790244,"statfs":{"total":386547056640,"available":382761209856,"internally_reserved":0,"allocated":1884160,"data_stored":896405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002511"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11993,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.360145+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:15.360145+0000","last_peered":"2024-09-15T03:19:15.360145+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:15.360145+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14913,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364839+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:15.364839+0000","last_peered":"2024-09-15T03:19:15.364839+0000","last_clean":"2024-09-15T03:19:15.364839+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:15.364839+0000","last_undegraded":"2024-09-15T03:19:15.364839+0000","last_fullsized":"2024-09-15T03:19:15.364839+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12345,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.365939+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:15.365939+0000","last_peered":"2024-09-15T03:19:15.365939+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:15.365939+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15615,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364298+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:15.364298+0000","last_peered":"2024-09-15T03:19:15.364298+0000","last_clean":"2024-09-15T03:19:15.364298+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:15.364298+0000","last_undegraded":"2024-09-15T03:19:15.364298+0000","last_fullsized":"2024-09-15T03:19:15.364298+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16387,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.359546+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:15.359546+0000","last_peered":"2024-09-15T03:19:15.359546+0000","last_clean":"2024-09-15T03:19:15.359546+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:15.359546+0000","last_undegraded":"2024-09-15T03:19:15.359546+0000","last_fullsized":"2024-09-15T03:19:15.359546+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12099,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.360132+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:15.360132+0000","last_peered":"2024-09-15T03:19:15.360132+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:15.360132+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16163,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.367148+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:15.367148+0000","last_peered":"2024-09-15T03:19:15.367148+0000","last_clean":"2024-09-15T03:19:15.367148+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:15.367148+0000","last_undegraded":"2024-09-15T03:19:15.367148+0000","last_fullsized":"2024-09-15T03:19:15.367148+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364479+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:15.364479+0000","last_peered":"2024-09-15T03:19:15.364479+0000","last_clean":"2024-09-15T03:19:15.364479+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:15.364479+0000","last_undegraded":"2024-09-15T03:19:15.364479+0000","last_fullsized":"2024-09-15T03:19:15.364479+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16147,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.359710+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:15.359710+0000","last_peered":"2024-09-15T03:19:15.359710+0000","last_clean":"2024-09-15T03:19:15.359710+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:15.359710+0000","last_undegraded":"2024-09-15T03:19:15.359710+0000","last_fullsized":"2024-09-15T03:19:15.359710+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16203,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.365705+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:15.365705+0000","last_peered":"2024-09-15T03:19:15.365705+0000","last_clean":"2024-09-15T03:19:15.365705+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:15.365705+0000","last_undegraded":"2024-09-15T03:19:15.365705+0000","last_fullsized":"2024-09-15T03:19:15.365705+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":12394,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.392286+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:27.392286+0000","last_peered":"2024-09-15T03:19:27.392286+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:27.392286+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":220,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1861}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15956,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364780+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:15.364780+0000","last_peered":"2024-09-15T03:19:15.364780+0000","last_clean":"2024-09-15T03:19:15.364780+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:15.364780+0000","last_undegraded":"2024-09-15T03:19:15.364780+0000","last_fullsized":"2024-09-15T03:19:15.364780+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16323,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.364612+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:15.364612+0000","last_peered":"2024-09-15T03:19:15.364612+0000","last_clean":"2024-09-15T03:19:15.364612+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:15.364612+0000","last_undegraded":"2024-09-15T03:19:15.364612+0000","last_fullsized":"2024-09-15T03:19:15.364612+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:15.359958+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:15.359958+0000","last_peered":"2024-09-15T03:19:15.359958+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:15.359958+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":11995,"reported_epoch":52,"state":"peering","last_fresh":"2024-09-15T03:19:26.411040+0000","last_change":"2024-09-15T03:19:26.410892+0000","last_active":"2024-09-15T03:19:26.410720+0000","last_peered":"2024-09-15T03:17:42.479969+0000","last_clean":"2024-09-15T03:05:05.401750+0000","last_became_active":"2024-09-15T03:06:09.251548+0000","last_became_peered":"2024-09-15T03:06:09.251548+0000","last_unstale":"2024-09-15T03:19:26.411040+0000","last_undegraded":"2024-09-15T03:19:26.411040+0000","last_fullsized":"2024-09-15T03:19:26.411040+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12138,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T03:19:15.458539+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:15.458539+0000","last_peered":"2024-09-15T03:19:15.458539+0000","last_clean":"2024-09-15T03:19:15.458539+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:15.458539+0000","last_undegraded":"2024-09-15T03:19:15.458539+0000","last_fullsized":"2024-09-15T03:19:15.458539+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20730,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932152,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439688,"statfs":{"total":96636764160,"available":95682240512,"internally_reserved":0,"allocated":438272,"data_stored":189431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.505}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.44400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919220,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452620,"statfs":{"total":96636764160,"available":95695482880,"internally_reserved":0,"allocated":434176,"data_stored":188481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.373}]},{"osd":1,"last update":"Sun Sep 15 03:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":2.3330000000000002,"15min":1.7829999999999999},"min":{"1min":0.26500000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":2.3370000000000002,"5min":15.833,"15min":32.295999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.85099999999999998,"5min":2.3889999999999998,"15min":1.845},"min":{"1min":0.33000000000000002,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":2.4039999999999999,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.54900000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.42099999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926072,"kb_used_data":560,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445768,"statfs":{"total":96636764160,"available":95688466432,"internally_reserved":0,"allocated":573440,"data_stored":329062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.64700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.747}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.59699999999999998}]}]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919672,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452168,"statfs":{"total":96636764160,"available":95695020032,"internally_reserved":0,"allocated":438272,"data_stored":189431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.55400000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.31},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.47299999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.503},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:35.506 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:35.814 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:35.816 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:35.826 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-15T03:19:35.025621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21560,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705480,"kb_used_data":1880,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781880,"statfs":{"total":386547056640,"available":382752645120,"internally_reserved":0,"allocated":1925120,"data_stored":914966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-289,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.623247"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11994,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.409111+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:26.409111+0000","last_peered":"2024-09-15T03:19:26.409111+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:26.409111+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14914,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.417966+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:26.417966+0000","last_peered":"2024-09-15T03:19:26.417966+0000","last_clean":"2024-09-15T03:19:26.417966+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:26.417966+0000","last_undegraded":"2024-09-15T03:19:26.417966+0000","last_fullsized":"2024-09-15T03:19:26.417966+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12346,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.405901+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:26.405901+0000","last_peered":"2024-09-15T03:19:26.405901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:26.405901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15616,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.410468+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:26.410468+0000","last_peered":"2024-09-15T03:19:26.410468+0000","last_clean":"2024-09-15T03:19:26.410468+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:26.410468+0000","last_undegraded":"2024-09-15T03:19:26.410468+0000","last_fullsized":"2024-09-15T03:19:26.410468+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16388,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407069+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:26.407069+0000","last_peered":"2024-09-15T03:19:26.407069+0000","last_clean":"2024-09-15T03:19:26.407069+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:26.407069+0000","last_undegraded":"2024-09-15T03:19:26.407069+0000","last_fullsized":"2024-09-15T03:19:26.407069+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12100,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.408249+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:26.408249+0000","last_peered":"2024-09-15T03:19:26.408249+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:26.408249+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.414304+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:26.414304+0000","last_peered":"2024-09-15T03:19:26.414304+0000","last_clean":"2024-09-15T03:19:26.414304+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:26.414304+0000","last_undegraded":"2024-09-15T03:19:26.414304+0000","last_fullsized":"2024-09-15T03:19:26.414304+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412194+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:26.412194+0000","last_peered":"2024-09-15T03:19:26.412194+0000","last_clean":"2024-09-15T03:19:26.412194+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:26.412194+0000","last_undegraded":"2024-09-15T03:19:26.412194+0000","last_fullsized":"2024-09-15T03:19:26.412194+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407100+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:26.407100+0000","last_peered":"2024-09-15T03:19:26.407100+0000","last_clean":"2024-09-15T03:19:26.407100+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:26.407100+0000","last_undegraded":"2024-09-15T03:19:26.407100+0000","last_fullsized":"2024-09-15T03:19:26.407100+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16204,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407328+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:26.407328+0000","last_peered":"2024-09-15T03:19:26.407328+0000","last_clean":"2024-09-15T03:19:26.407328+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:26.407328+0000","last_undegraded":"2024-09-15T03:19:26.407328+0000","last_fullsized":"2024-09-15T03:19:26.407328+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":14055,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:32.397059+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:32.397059+0000","last_peered":"2024-09-15T03:19:32.397059+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:32.397059+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15957,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412422+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:26.412422+0000","last_peered":"2024-09-15T03:19:26.412422+0000","last_clean":"2024-09-15T03:19:26.412422+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:26.412422+0000","last_undegraded":"2024-09-15T03:19:26.412422+0000","last_fullsized":"2024-09-15T03:19:26.412422+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.411816+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:26.411816+0000","last_peered":"2024-09-15T03:19:26.411816+0000","last_clean":"2024-09-15T03:19:26.411816+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:26.411816+0000","last_undegraded":"2024-09-15T03:19:26.411816+0000","last_fullsized":"2024-09-15T03:19:26.411816+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.407655+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:26.407655+0000","last_peered":"2024-09-15T03:19:26.407655+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:26.407655+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12005,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.504838+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.504838+0000","last_peered":"2024-09-15T03:19:27.504838+0000","last_clean":"2024-09-15T03:19:27.504838+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.504838+0000","last_undegraded":"2024-09-15T03:19:27.504838+0000","last_fullsized":"2024-09-15T03:19:27.504838+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412296+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:26.412296+0000","last_peered":"2024-09-15T03:19:26.412296+0000","last_clean":"2024-09-15T03:19:26.412296+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:26.412296+0000","last_undegraded":"2024-09-15T03:19:26.412296+0000","last_fullsized":"2024-09-15T03:19:26.412296+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21560,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932160,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439680,"statfs":{"total":96636764160,"available":95682232320,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.627}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919232,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452608,"statfs":{"total":96636764160,"available":95695470592,"internally_reserved":0,"allocated":442368,"data_stored":192189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.84599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.89100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.73799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934408,"kb_used_data":576,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437432,"statfs":{"total":96636764160,"available":95679930368,"internally_reserved":0,"allocated":589824,"data_stored":336499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.46999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.374},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.44400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919680,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452160,"statfs":{"total":96636764160,"available":95695011840,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:35.828 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:36.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-15T03:19:36.146+0000 7f39dbfff700 0 --2- 172.21.15.60:0/2131954543 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39dc06c6b0 0x7f39dc124750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-15T03:19:36.363 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:36.366 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:36.380 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-15T03:19:35.025621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21560,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705480,"kb_used_data":1880,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781880,"statfs":{"total":386547056640,"available":382752645120,"internally_reserved":0,"allocated":1925120,"data_stored":914966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-289,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.623247"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11994,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.409111+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:26.409111+0000","last_peered":"2024-09-15T03:19:26.409111+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:26.409111+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14914,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.417966+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:26.417966+0000","last_peered":"2024-09-15T03:19:26.417966+0000","last_clean":"2024-09-15T03:19:26.417966+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:26.417966+0000","last_undegraded":"2024-09-15T03:19:26.417966+0000","last_fullsized":"2024-09-15T03:19:26.417966+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12346,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.405901+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:26.405901+0000","last_peered":"2024-09-15T03:19:26.405901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:26.405901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15616,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.410468+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:26.410468+0000","last_peered":"2024-09-15T03:19:26.410468+0000","last_clean":"2024-09-15T03:19:26.410468+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:26.410468+0000","last_undegraded":"2024-09-15T03:19:26.410468+0000","last_fullsized":"2024-09-15T03:19:26.410468+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16388,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407069+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:26.407069+0000","last_peered":"2024-09-15T03:19:26.407069+0000","last_clean":"2024-09-15T03:19:26.407069+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:26.407069+0000","last_undegraded":"2024-09-15T03:19:26.407069+0000","last_fullsized":"2024-09-15T03:19:26.407069+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12100,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.408249+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:26.408249+0000","last_peered":"2024-09-15T03:19:26.408249+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:26.408249+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.414304+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:26.414304+0000","last_peered":"2024-09-15T03:19:26.414304+0000","last_clean":"2024-09-15T03:19:26.414304+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:26.414304+0000","last_undegraded":"2024-09-15T03:19:26.414304+0000","last_fullsized":"2024-09-15T03:19:26.414304+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412194+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:26.412194+0000","last_peered":"2024-09-15T03:19:26.412194+0000","last_clean":"2024-09-15T03:19:26.412194+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:26.412194+0000","last_undegraded":"2024-09-15T03:19:26.412194+0000","last_fullsized":"2024-09-15T03:19:26.412194+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407100+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:26.407100+0000","last_peered":"2024-09-15T03:19:26.407100+0000","last_clean":"2024-09-15T03:19:26.407100+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:26.407100+0000","last_undegraded":"2024-09-15T03:19:26.407100+0000","last_fullsized":"2024-09-15T03:19:26.407100+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16204,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407328+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:26.407328+0000","last_peered":"2024-09-15T03:19:26.407328+0000","last_clean":"2024-09-15T03:19:26.407328+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:26.407328+0000","last_undegraded":"2024-09-15T03:19:26.407328+0000","last_fullsized":"2024-09-15T03:19:26.407328+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":14055,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:32.397059+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:32.397059+0000","last_peered":"2024-09-15T03:19:32.397059+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:32.397059+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15957,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412422+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:26.412422+0000","last_peered":"2024-09-15T03:19:26.412422+0000","last_clean":"2024-09-15T03:19:26.412422+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:26.412422+0000","last_undegraded":"2024-09-15T03:19:26.412422+0000","last_fullsized":"2024-09-15T03:19:26.412422+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.411816+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:26.411816+0000","last_peered":"2024-09-15T03:19:26.411816+0000","last_clean":"2024-09-15T03:19:26.411816+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:26.411816+0000","last_undegraded":"2024-09-15T03:19:26.411816+0000","last_fullsized":"2024-09-15T03:19:26.411816+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.407655+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:26.407655+0000","last_peered":"2024-09-15T03:19:26.407655+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:26.407655+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12005,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.504838+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.504838+0000","last_peered":"2024-09-15T03:19:27.504838+0000","last_clean":"2024-09-15T03:19:27.504838+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.504838+0000","last_undegraded":"2024-09-15T03:19:27.504838+0000","last_fullsized":"2024-09-15T03:19:27.504838+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412296+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:26.412296+0000","last_peered":"2024-09-15T03:19:26.412296+0000","last_clean":"2024-09-15T03:19:26.412296+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:26.412296+0000","last_undegraded":"2024-09-15T03:19:26.412296+0000","last_fullsized":"2024-09-15T03:19:26.412296+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21560,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932160,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439680,"statfs":{"total":96636764160,"available":95682232320,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.627}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919232,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452608,"statfs":{"total":96636764160,"available":95695470592,"internally_reserved":0,"allocated":442368,"data_stored":192189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.84599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.89100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.73799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934408,"kb_used_data":576,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437432,"statfs":{"total":96636764160,"available":95679930368,"internally_reserved":0,"allocated":589824,"data_stored":336499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.52900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.46999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.374},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.44400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919680,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452160,"statfs":{"total":96636764160,"available":95695011840,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:36.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:36.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11994, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:26.409111+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:26.409111+0000', 'last_peered': '2024-09-15T03:19:26.409111+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:26.409111+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:36.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:36.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12346, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:26.405901+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:19:26.405901+0000', 'last_peered': '2024-09-15T03:19:26.405901+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:26.405901+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:36.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:36.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12100, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:26.408249+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:26.408249+0000', 'last_peered': '2024-09-15T03:19:26.408249+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:26.408249+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:36.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:36.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 14055, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:32.397059+0000', 'last_change': '2024-09-15T03:19:25.848067+0000', 'last_active': '2024-09-15T03:19:32.397059+0000', 'last_peered': '2024-09-15T03:19:32.397059+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:32.397059+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1034, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1050, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:36.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:36.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11798, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:26.407655+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:26.407655+0000', 'last_peered': '2024-09-15T03:19:26.407655+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:26.407655+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:36.382 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:36.755 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:36.755 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9145/112 objects degraded (8165.179%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":899,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3794411520,"bytes_avail":382752645120,"bytes_total":386547056640,"degraded_objects":9145,"degraded_total":112,"degraded_ratio":81.651785714285708},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:36.773 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T03:19:36.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:37.084 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:37.089 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:37.102 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-15T03:19:37.026219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21560,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693000,"kb_used_data":1880,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794360,"statfs":{"total":386547056640,"available":382765424640,"internally_reserved":0,"allocated":1925120,"data_stored":914966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003060"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11994,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.409111+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:26.409111+0000","last_peered":"2024-09-15T03:19:26.409111+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:26.409111+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14914,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.417966+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:26.417966+0000","last_peered":"2024-09-15T03:19:26.417966+0000","last_clean":"2024-09-15T03:19:26.417966+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:26.417966+0000","last_undegraded":"2024-09-15T03:19:26.417966+0000","last_fullsized":"2024-09-15T03:19:26.417966+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12346,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.405901+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:26.405901+0000","last_peered":"2024-09-15T03:19:26.405901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:26.405901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15616,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.410468+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:26.410468+0000","last_peered":"2024-09-15T03:19:26.410468+0000","last_clean":"2024-09-15T03:19:26.410468+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:26.410468+0000","last_undegraded":"2024-09-15T03:19:26.410468+0000","last_fullsized":"2024-09-15T03:19:26.410468+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16388,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407069+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:26.407069+0000","last_peered":"2024-09-15T03:19:26.407069+0000","last_clean":"2024-09-15T03:19:26.407069+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:26.407069+0000","last_undegraded":"2024-09-15T03:19:26.407069+0000","last_fullsized":"2024-09-15T03:19:26.407069+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12100,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.408249+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:26.408249+0000","last_peered":"2024-09-15T03:19:26.408249+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:26.408249+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.414304+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:26.414304+0000","last_peered":"2024-09-15T03:19:26.414304+0000","last_clean":"2024-09-15T03:19:26.414304+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:26.414304+0000","last_undegraded":"2024-09-15T03:19:26.414304+0000","last_fullsized":"2024-09-15T03:19:26.414304+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412194+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:26.412194+0000","last_peered":"2024-09-15T03:19:26.412194+0000","last_clean":"2024-09-15T03:19:26.412194+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:26.412194+0000","last_undegraded":"2024-09-15T03:19:26.412194+0000","last_fullsized":"2024-09-15T03:19:26.412194+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407100+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:26.407100+0000","last_peered":"2024-09-15T03:19:26.407100+0000","last_clean":"2024-09-15T03:19:26.407100+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:26.407100+0000","last_undegraded":"2024-09-15T03:19:26.407100+0000","last_fullsized":"2024-09-15T03:19:26.407100+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16204,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407328+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:26.407328+0000","last_peered":"2024-09-15T03:19:26.407328+0000","last_clean":"2024-09-15T03:19:26.407328+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:26.407328+0000","last_undegraded":"2024-09-15T03:19:26.407328+0000","last_fullsized":"2024-09-15T03:19:26.407328+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":14055,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:32.397059+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:32.397059+0000","last_peered":"2024-09-15T03:19:32.397059+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:32.397059+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1034,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15957,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412422+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:26.412422+0000","last_peered":"2024-09-15T03:19:26.412422+0000","last_clean":"2024-09-15T03:19:26.412422+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:26.412422+0000","last_undegraded":"2024-09-15T03:19:26.412422+0000","last_fullsized":"2024-09-15T03:19:26.412422+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.411816+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:26.411816+0000","last_peered":"2024-09-15T03:19:26.411816+0000","last_clean":"2024-09-15T03:19:26.411816+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:26.411816+0000","last_undegraded":"2024-09-15T03:19:26.411816+0000","last_fullsized":"2024-09-15T03:19:26.411816+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.407655+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:26.407655+0000","last_peered":"2024-09-15T03:19:26.407655+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:26.407655+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12005,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.504838+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.504838+0000","last_peered":"2024-09-15T03:19:27.504838+0000","last_clean":"2024-09-15T03:19:27.504838+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.504838+0000","last_undegraded":"2024-09-15T03:19:27.504838+0000","last_fullsized":"2024-09-15T03:19:27.504838+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412296+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:26.412296+0000","last_peered":"2024-09-15T03:19:26.412296+0000","last_clean":"2024-09-15T03:19:26.412296+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:26.412296+0000","last_undegraded":"2024-09-15T03:19:26.412296+0000","last_fullsized":"2024-09-15T03:19:26.412296+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21560,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932160,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439680,"statfs":{"total":96636764160,"available":95682232320,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.627}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919232,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452608,"statfs":{"total":96636764160,"available":95695470592,"internally_reserved":0,"allocated":442368,"data_stored":192189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.84599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.89100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.73799999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921928,"kb_used_data":576,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449912,"statfs":{"total":96636764160,"available":95692709888,"internally_reserved":0,"allocated":589824,"data_stored":336499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.45700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.55600000000000005}]}]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919680,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452160,"statfs":{"total":96636764160,"available":95695011840,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:40.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:40.421 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:40.424 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:40.436 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-15T03:19:39.026754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22154,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693000,"kb_used_data":1880,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794360,"statfs":{"total":386547056640,"available":382765424640,"internally_reserved":0,"allocated":1925120,"data_stored":914966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.602235"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11994,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.409111+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:26.409111+0000","last_peered":"2024-09-15T03:19:26.409111+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:26.409111+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14914,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.417966+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:26.417966+0000","last_peered":"2024-09-15T03:19:26.417966+0000","last_clean":"2024-09-15T03:19:26.417966+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:26.417966+0000","last_undegraded":"2024-09-15T03:19:26.417966+0000","last_fullsized":"2024-09-15T03:19:26.417966+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12346,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.405901+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:26.405901+0000","last_peered":"2024-09-15T03:19:26.405901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:26.405901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15616,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.410468+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:26.410468+0000","last_peered":"2024-09-15T03:19:26.410468+0000","last_clean":"2024-09-15T03:19:26.410468+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:26.410468+0000","last_undegraded":"2024-09-15T03:19:26.410468+0000","last_fullsized":"2024-09-15T03:19:26.410468+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16388,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407069+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:26.407069+0000","last_peered":"2024-09-15T03:19:26.407069+0000","last_clean":"2024-09-15T03:19:26.407069+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:26.407069+0000","last_undegraded":"2024-09-15T03:19:26.407069+0000","last_fullsized":"2024-09-15T03:19:26.407069+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12100,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.408249+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:26.408249+0000","last_peered":"2024-09-15T03:19:26.408249+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:26.408249+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.414304+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:26.414304+0000","last_peered":"2024-09-15T03:19:26.414304+0000","last_clean":"2024-09-15T03:19:26.414304+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:26.414304+0000","last_undegraded":"2024-09-15T03:19:26.414304+0000","last_fullsized":"2024-09-15T03:19:26.414304+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412194+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:26.412194+0000","last_peered":"2024-09-15T03:19:26.412194+0000","last_clean":"2024-09-15T03:19:26.412194+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:26.412194+0000","last_undegraded":"2024-09-15T03:19:26.412194+0000","last_fullsized":"2024-09-15T03:19:26.412194+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407100+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:26.407100+0000","last_peered":"2024-09-15T03:19:26.407100+0000","last_clean":"2024-09-15T03:19:26.407100+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:26.407100+0000","last_undegraded":"2024-09-15T03:19:26.407100+0000","last_fullsized":"2024-09-15T03:19:26.407100+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16204,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407328+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:26.407328+0000","last_peered":"2024-09-15T03:19:26.407328+0000","last_clean":"2024-09-15T03:19:26.407328+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:26.407328+0000","last_undegraded":"2024-09-15T03:19:26.407328+0000","last_fullsized":"2024-09-15T03:19:26.407328+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":15243,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:37.394501+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:37.394501+0000","last_peered":"2024-09-15T03:19:37.394501+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:37.394501+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15957,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412422+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:26.412422+0000","last_peered":"2024-09-15T03:19:26.412422+0000","last_clean":"2024-09-15T03:19:26.412422+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:26.412422+0000","last_undegraded":"2024-09-15T03:19:26.412422+0000","last_fullsized":"2024-09-15T03:19:26.412422+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.411816+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:26.411816+0000","last_peered":"2024-09-15T03:19:26.411816+0000","last_clean":"2024-09-15T03:19:26.411816+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:26.411816+0000","last_undegraded":"2024-09-15T03:19:26.411816+0000","last_fullsized":"2024-09-15T03:19:26.411816+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.407655+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:26.407655+0000","last_peered":"2024-09-15T03:19:26.407655+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:26.407655+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12005,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.504838+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.504838+0000","last_peered":"2024-09-15T03:19:27.504838+0000","last_clean":"2024-09-15T03:19:27.504838+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.504838+0000","last_undegraded":"2024-09-15T03:19:27.504838+0000","last_fullsized":"2024-09-15T03:19:27.504838+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412296+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:26.412296+0000","last_peered":"2024-09-15T03:19:26.412296+0000","last_clean":"2024-09-15T03:19:26.412296+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:26.412296+0000","last_undegraded":"2024-09-15T03:19:26.412296+0000","last_fullsized":"2024-09-15T03:19:26.412296+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22154,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932160,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439680,"statfs":{"total":96636764160,"available":95682232320,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.438},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.58499999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.77100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.82499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919232,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452608,"statfs":{"total":96636764160,"available":95695470592,"internally_reserved":0,"allocated":442368,"data_stored":192189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.48199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.52200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921928,"kb_used_data":576,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449912,"statfs":{"total":96636764160,"available":95692709888,"internally_reserved":0,"allocated":589824,"data_stored":336499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.45700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.55600000000000005}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919680,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452160,"statfs":{"total":96636764160,"available":95695011840,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:40.438 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:40.756 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:40.758 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:40.769 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-09-15T03:19:39.026754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22154,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693000,"kb_used_data":1880,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794360,"statfs":{"total":386547056640,"available":382765424640,"internally_reserved":0,"allocated":1925120,"data_stored":914966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.602235"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11994,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.409111+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:26.409111+0000","last_peered":"2024-09-15T03:19:26.409111+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:26.409111+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14914,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.417966+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:26.417966+0000","last_peered":"2024-09-15T03:19:26.417966+0000","last_clean":"2024-09-15T03:19:26.417966+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:26.417966+0000","last_undegraded":"2024-09-15T03:19:26.417966+0000","last_fullsized":"2024-09-15T03:19:26.417966+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12346,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.405901+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:26.405901+0000","last_peered":"2024-09-15T03:19:26.405901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:26.405901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15616,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.410468+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:26.410468+0000","last_peered":"2024-09-15T03:19:26.410468+0000","last_clean":"2024-09-15T03:19:26.410468+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:26.410468+0000","last_undegraded":"2024-09-15T03:19:26.410468+0000","last_fullsized":"2024-09-15T03:19:26.410468+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16388,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407069+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:26.407069+0000","last_peered":"2024-09-15T03:19:26.407069+0000","last_clean":"2024-09-15T03:19:26.407069+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:26.407069+0000","last_undegraded":"2024-09-15T03:19:26.407069+0000","last_fullsized":"2024-09-15T03:19:26.407069+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12100,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.408249+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:26.408249+0000","last_peered":"2024-09-15T03:19:26.408249+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:26.408249+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.414304+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:26.414304+0000","last_peered":"2024-09-15T03:19:26.414304+0000","last_clean":"2024-09-15T03:19:26.414304+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:26.414304+0000","last_undegraded":"2024-09-15T03:19:26.414304+0000","last_fullsized":"2024-09-15T03:19:26.414304+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412194+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:26.412194+0000","last_peered":"2024-09-15T03:19:26.412194+0000","last_clean":"2024-09-15T03:19:26.412194+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:26.412194+0000","last_undegraded":"2024-09-15T03:19:26.412194+0000","last_fullsized":"2024-09-15T03:19:26.412194+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407100+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:26.407100+0000","last_peered":"2024-09-15T03:19:26.407100+0000","last_clean":"2024-09-15T03:19:26.407100+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:26.407100+0000","last_undegraded":"2024-09-15T03:19:26.407100+0000","last_fullsized":"2024-09-15T03:19:26.407100+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16204,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407328+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:26.407328+0000","last_peered":"2024-09-15T03:19:26.407328+0000","last_clean":"2024-09-15T03:19:26.407328+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:26.407328+0000","last_undegraded":"2024-09-15T03:19:26.407328+0000","last_fullsized":"2024-09-15T03:19:26.407328+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":15243,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:37.394501+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:37.394501+0000","last_peered":"2024-09-15T03:19:37.394501+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:37.394501+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15957,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412422+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:26.412422+0000","last_peered":"2024-09-15T03:19:26.412422+0000","last_clean":"2024-09-15T03:19:26.412422+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:26.412422+0000","last_undegraded":"2024-09-15T03:19:26.412422+0000","last_fullsized":"2024-09-15T03:19:26.412422+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.411816+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:26.411816+0000","last_peered":"2024-09-15T03:19:26.411816+0000","last_clean":"2024-09-15T03:19:26.411816+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:26.411816+0000","last_undegraded":"2024-09-15T03:19:26.411816+0000","last_fullsized":"2024-09-15T03:19:26.411816+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.407655+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:26.407655+0000","last_peered":"2024-09-15T03:19:26.407655+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:26.407655+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12005,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.504838+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.504838+0000","last_peered":"2024-09-15T03:19:27.504838+0000","last_clean":"2024-09-15T03:19:27.504838+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.504838+0000","last_undegraded":"2024-09-15T03:19:27.504838+0000","last_fullsized":"2024-09-15T03:19:27.504838+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412296+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:26.412296+0000","last_peered":"2024-09-15T03:19:26.412296+0000","last_clean":"2024-09-15T03:19:26.412296+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:26.412296+0000","last_undegraded":"2024-09-15T03:19:26.412296+0000","last_fullsized":"2024-09-15T03:19:26.412296+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22154,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932160,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439680,"statfs":{"total":96636764160,"available":95682232320,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.438},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.58499999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.77100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.82499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919232,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452608,"statfs":{"total":96636764160,"available":95695470592,"internally_reserved":0,"allocated":442368,"data_stored":192189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.48199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.52200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921928,"kb_used_data":576,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449912,"statfs":{"total":96636764160,"available":95692709888,"internally_reserved":0,"allocated":589824,"data_stored":336499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.45700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.55600000000000005}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919680,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452160,"statfs":{"total":96636764160,"available":95695011840,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:40.770 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11994, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:26.409111+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:26.409111+0000', 'last_peered': '2024-09-15T03:19:26.409111+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:26.409111+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12346, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:26.405901+0000', 'last_change': '2024-09-15T03:06:09.331695+0000', 'last_active': '2024-09-15T03:19:26.405901+0000', 'last_peered': '2024-09-15T03:19:26.405901+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:26.405901+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2090}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12100, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:26.408249+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:26.408249+0000', 'last_peered': '2024-09-15T03:19:26.408249+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:26.408249+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 15243, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:37.394501+0000', 'last_change': '2024-09-15T03:19:25.848067+0000', 'last_active': '2024-09-15T03:19:37.394501+0000', 'last_peered': '2024-09-15T03:19:37.394501+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:37.394501+0000', 'last_undegraded': '2024-09-15T03:06:08.927129+0000', 'last_fullsized': '2024-09-15T03:06:08.836550+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 440, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1644, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -437}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:40.771 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11798, 'reported_epoch': 52, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:26.407655+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:26.407655+0000', 'last_peered': '2024-09-15T03:19:26.407655+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:26.407655+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:40.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:41.120 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:41.121 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8551/112 objects degraded (7634.821%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":904,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3781632000,"bytes_avail":382765424640,"bytes_total":386547056640,"degraded_objects":8551,"degraded_total":112,"degraded_ratio":76.348214285714292,"recovering_objects_per_sec":122,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1424,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:41.136 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:41.136 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:41.431 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:41.434 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:41.445 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-15T03:19:41.027652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22154,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693000,"kb_used_data":1880,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794360,"statfs":{"total":386547056640,"available":382765424640,"internally_reserved":0,"allocated":1925120,"data_stored":914966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1424,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003931"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11994,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.409111+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:26.409111+0000","last_peered":"2024-09-15T03:19:26.409111+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:26.409111+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14914,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.417966+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:26.417966+0000","last_peered":"2024-09-15T03:19:26.417966+0000","last_clean":"2024-09-15T03:19:26.417966+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:26.417966+0000","last_undegraded":"2024-09-15T03:19:26.417966+0000","last_fullsized":"2024-09-15T03:19:26.417966+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12346,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.405901+0000","last_change":"2024-09-15T03:06:09.331695+0000","last_active":"2024-09-15T03:19:26.405901+0000","last_peered":"2024-09-15T03:19:26.405901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:26.405901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15616,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.410468+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:26.410468+0000","last_peered":"2024-09-15T03:19:26.410468+0000","last_clean":"2024-09-15T03:19:26.410468+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:26.410468+0000","last_undegraded":"2024-09-15T03:19:26.410468+0000","last_fullsized":"2024-09-15T03:19:26.410468+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16388,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407069+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:26.407069+0000","last_peered":"2024-09-15T03:19:26.407069+0000","last_clean":"2024-09-15T03:19:26.407069+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:26.407069+0000","last_undegraded":"2024-09-15T03:19:26.407069+0000","last_fullsized":"2024-09-15T03:19:26.407069+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12100,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.408249+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:26.408249+0000","last_peered":"2024-09-15T03:19:26.408249+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:26.408249+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16164,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.414304+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:26.414304+0000","last_peered":"2024-09-15T03:19:26.414304+0000","last_clean":"2024-09-15T03:19:26.414304+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:26.414304+0000","last_undegraded":"2024-09-15T03:19:26.414304+0000","last_fullsized":"2024-09-15T03:19:26.414304+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12227,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412194+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:26.412194+0000","last_peered":"2024-09-15T03:19:26.412194+0000","last_clean":"2024-09-15T03:19:26.412194+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:26.412194+0000","last_undegraded":"2024-09-15T03:19:26.412194+0000","last_fullsized":"2024-09-15T03:19:26.412194+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16148,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407100+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:26.407100+0000","last_peered":"2024-09-15T03:19:26.407100+0000","last_clean":"2024-09-15T03:19:26.407100+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:26.407100+0000","last_undegraded":"2024-09-15T03:19:26.407100+0000","last_fullsized":"2024-09-15T03:19:26.407100+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16204,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.407328+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:26.407328+0000","last_peered":"2024-09-15T03:19:26.407328+0000","last_clean":"2024-09-15T03:19:26.407328+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:26.407328+0000","last_undegraded":"2024-09-15T03:19:26.407328+0000","last_fullsized":"2024-09-15T03:19:26.407328+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":15243,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:37.394501+0000","last_change":"2024-09-15T03:19:25.848067+0000","last_active":"2024-09-15T03:19:37.394501+0000","last_peered":"2024-09-15T03:19:37.394501+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:37.394501+0000","last_undegraded":"2024-09-15T03:06:08.927129+0000","last_fullsized":"2024-09-15T03:06:08.836550+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":440,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1644,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-437}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15957,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412422+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:26.412422+0000","last_peered":"2024-09-15T03:19:26.412422+0000","last_clean":"2024-09-15T03:19:26.412422+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:26.412422+0000","last_undegraded":"2024-09-15T03:19:26.412422+0000","last_fullsized":"2024-09-15T03:19:26.412422+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16324,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.411816+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:26.411816+0000","last_peered":"2024-09-15T03:19:26.411816+0000","last_clean":"2024-09-15T03:19:26.411816+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:26.411816+0000","last_undegraded":"2024-09-15T03:19:26.411816+0000","last_fullsized":"2024-09-15T03:19:26.411816+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":52,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:26.407655+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:26.407655+0000","last_peered":"2024-09-15T03:19:26.407655+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:26.407655+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12005,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.504838+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.504838+0000","last_peered":"2024-09-15T03:19:27.504838+0000","last_clean":"2024-09-15T03:19:27.504838+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.504838+0000","last_undegraded":"2024-09-15T03:19:27.504838+0000","last_fullsized":"2024-09-15T03:19:27.504838+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12139,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-15T03:19:26.412296+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:26.412296+0000","last_peered":"2024-09-15T03:19:26.412296+0000","last_clean":"2024-09-15T03:19:26.412296+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:26.412296+0000","last_undegraded":"2024-09-15T03:19:26.412296+0000","last_fullsized":"2024-09-15T03:19:26.412296+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22154,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":159744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932160,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439680,"statfs":{"total":96636764160,"available":95682232320,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.438},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.58499999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:18:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8420000000000001,"5min":4.1779999999999999,"15min":2.9409999999999998},"min":{"1min":0.32300000000000001,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":22.928999999999998,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":2.8279999999999998,"15min":1.9750000000000001},"min":{"1min":0.33300000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":0.76900000000000002,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.77100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.82499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919232,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452608,"statfs":{"total":96636764160,"available":95695470592,"internally_reserved":0,"allocated":442368,"data_stored":192189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.48199999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.52200000000000002}]}]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921928,"kb_used_data":576,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449912,"statfs":{"total":96636764160,"available":95692709888,"internally_reserved":0,"allocated":589824,"data_stored":336499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.45700000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.55600000000000005}]}]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919680,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452160,"statfs":{"total":96636764160,"available":95695011840,"internally_reserved":0,"allocated":446464,"data_stored":193139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:18:33 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":2.1659999999999999,"15min":1.708},"min":{"1min":0.27000000000000002,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":21.646999999999998,"5min":22.384,"15min":22.384},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":3.044,"15min":2.649},"min":{"1min":0.26200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":18.344000000000001,"5min":30.433,"15min":30.433},"last":0.42699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:44.447 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:44.769 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:44.771 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:44.782 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-15T03:19:43.028100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22786,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697176,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790184,"statfs":{"total":386547056640,"available":382761148416,"internally_reserved":0,"allocated":1945600,"data_stored":921946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002478"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11995,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.423186+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:27.423186+0000","last_peered":"2024-09-15T03:19:27.423186+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:27.423186+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14915,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.423070+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:27.423070+0000","last_peered":"2024-09-15T03:19:27.423070+0000","last_clean":"2024-09-15T03:19:27.423070+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:27.423070+0000","last_undegraded":"2024-09-15T03:19:27.423070+0000","last_fullsized":"2024-09-15T03:19:27.423070+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12732,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:42.222901+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:42.222901+0000","last_peered":"2024-09-15T03:19:42.222901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:42.222901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1899}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15617,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419628+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:27.419628+0000","last_peered":"2024-09-15T03:19:27.419628+0000","last_clean":"2024-09-15T03:19:27.419628+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:27.419628+0000","last_undegraded":"2024-09-15T03:19:27.419628+0000","last_fullsized":"2024-09-15T03:19:27.419628+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16389,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419275+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:27.419275+0000","last_peered":"2024-09-15T03:19:27.419275+0000","last_clean":"2024-09-15T03:19:27.419275+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:27.419275+0000","last_undegraded":"2024-09-15T03:19:27.419275+0000","last_fullsized":"2024-09-15T03:19:27.419275+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12101,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.423090+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:27.423090+0000","last_peered":"2024-09-15T03:19:27.423090+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:27.423090+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16165,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.426280+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:27.426280+0000","last_peered":"2024-09-15T03:19:27.426280+0000","last_clean":"2024-09-15T03:19:27.426280+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:27.426280+0000","last_undegraded":"2024-09-15T03:19:27.426280+0000","last_fullsized":"2024-09-15T03:19:27.426280+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12228,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419604+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:27.419604+0000","last_peered":"2024-09-15T03:19:27.419604+0000","last_clean":"2024-09-15T03:19:27.419604+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:27.419604+0000","last_undegraded":"2024-09-15T03:19:27.419604+0000","last_fullsized":"2024-09-15T03:19:27.419604+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16149,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.422621+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:27.422621+0000","last_peered":"2024-09-15T03:19:27.422621+0000","last_clean":"2024-09-15T03:19:27.422621+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:27.422621+0000","last_undegraded":"2024-09-15T03:19:27.422621+0000","last_fullsized":"2024-09-15T03:19:27.422621+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16205,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.421132+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:27.421132+0000","last_peered":"2024-09-15T03:19:27.421132+0000","last_clean":"2024-09-15T03:19:27.421132+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:27.421132+0000","last_undegraded":"2024-09-15T03:19:27.421132+0000","last_fullsized":"2024-09-15T03:19:27.421132+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16127,"reported_epoch":54,"state":"peering","last_fresh":"2024-09-15T03:19:41.453724+0000","last_change":"2024-09-15T03:19:41.457283+0000","last_active":"2024-09-15T03:19:41.453683+0000","last_peered":"2024-09-15T03:19:40.664154+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:41.453724+0000","last_undegraded":"2024-09-15T03:19:41.453724+0000","last_fullsized":"2024-09-15T03:19:41.453724+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15958,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419887+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:27.419887+0000","last_peered":"2024-09-15T03:19:27.419887+0000","last_clean":"2024-09-15T03:19:27.419887+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:27.419887+0000","last_undegraded":"2024-09-15T03:19:27.419887+0000","last_fullsized":"2024-09-15T03:19:27.419887+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16325,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419752+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:27.419752+0000","last_peered":"2024-09-15T03:19:27.419752+0000","last_clean":"2024-09-15T03:19:27.419752+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:27.419752+0000","last_undegraded":"2024-09-15T03:19:27.419752+0000","last_fullsized":"2024-09-15T03:19:27.419752+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.419754+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:27.419754+0000","last_peered":"2024-09-15T03:19:27.419754+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:27.419754+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12006,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.505878+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.505878+0000","last_peered":"2024-09-15T03:19:27.505878+0000","last_clean":"2024-09-15T03:19:27.505878+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.505878+0000","last_undegraded":"2024-09-15T03:19:27.505878+0000","last_fullsized":"2024-09-15T03:19:27.505878+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12140,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419978+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:27.419978+0000","last_peered":"2024-09-15T03:19:27.419978+0000","last_clean":"2024-09-15T03:19:27.419978+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:27.419978+0000","last_undegraded":"2024-09-15T03:19:27.419978+0000","last_fullsized":"2024-09-15T03:19:27.419978+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22786,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932168,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439672,"statfs":{"total":96636764160,"available":95682224128,"internally_reserved":0,"allocated":454656,"data_stored":196831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.504}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":450560,"data_stored":195881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.439},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63300000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.47399999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926084,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445756,"statfs":{"total":96636764160,"available":95688454144,"internally_reserved":0,"allocated":585728,"data_stored":332403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.80600000000000005}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919688,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452152,"statfs":{"total":96636764160,"available":95695003648,"internally_reserved":0,"allocated":454656,"data_stored":196831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.67400000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56799999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:44.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:45.090 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:45.093 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:45.107 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-15T03:19:45.028512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22786,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697176,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790184,"statfs":{"total":386547056640,"available":382761148416,"internally_reserved":0,"allocated":1945600,"data_stored":921946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002293"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11995,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.423186+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:27.423186+0000","last_peered":"2024-09-15T03:19:27.423186+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:27.423186+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14915,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.423070+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:27.423070+0000","last_peered":"2024-09-15T03:19:27.423070+0000","last_clean":"2024-09-15T03:19:27.423070+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:27.423070+0000","last_undegraded":"2024-09-15T03:19:27.423070+0000","last_fullsized":"2024-09-15T03:19:27.423070+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12732,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:42.222901+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:42.222901+0000","last_peered":"2024-09-15T03:19:42.222901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:42.222901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1899}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15617,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419628+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:27.419628+0000","last_peered":"2024-09-15T03:19:27.419628+0000","last_clean":"2024-09-15T03:19:27.419628+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:27.419628+0000","last_undegraded":"2024-09-15T03:19:27.419628+0000","last_fullsized":"2024-09-15T03:19:27.419628+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16389,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419275+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:27.419275+0000","last_peered":"2024-09-15T03:19:27.419275+0000","last_clean":"2024-09-15T03:19:27.419275+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:27.419275+0000","last_undegraded":"2024-09-15T03:19:27.419275+0000","last_fullsized":"2024-09-15T03:19:27.419275+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12101,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.423090+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:27.423090+0000","last_peered":"2024-09-15T03:19:27.423090+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:27.423090+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16165,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.426280+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:27.426280+0000","last_peered":"2024-09-15T03:19:27.426280+0000","last_clean":"2024-09-15T03:19:27.426280+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:27.426280+0000","last_undegraded":"2024-09-15T03:19:27.426280+0000","last_fullsized":"2024-09-15T03:19:27.426280+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12228,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419604+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:27.419604+0000","last_peered":"2024-09-15T03:19:27.419604+0000","last_clean":"2024-09-15T03:19:27.419604+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:27.419604+0000","last_undegraded":"2024-09-15T03:19:27.419604+0000","last_fullsized":"2024-09-15T03:19:27.419604+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16149,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.422621+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:27.422621+0000","last_peered":"2024-09-15T03:19:27.422621+0000","last_clean":"2024-09-15T03:19:27.422621+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:27.422621+0000","last_undegraded":"2024-09-15T03:19:27.422621+0000","last_fullsized":"2024-09-15T03:19:27.422621+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16205,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.421132+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:27.421132+0000","last_peered":"2024-09-15T03:19:27.421132+0000","last_clean":"2024-09-15T03:19:27.421132+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:27.421132+0000","last_undegraded":"2024-09-15T03:19:27.421132+0000","last_fullsized":"2024-09-15T03:19:27.421132+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16127,"reported_epoch":54,"state":"peering","last_fresh":"2024-09-15T03:19:41.453724+0000","last_change":"2024-09-15T03:19:41.457283+0000","last_active":"2024-09-15T03:19:41.453683+0000","last_peered":"2024-09-15T03:19:40.664154+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:41.453724+0000","last_undegraded":"2024-09-15T03:19:41.453724+0000","last_fullsized":"2024-09-15T03:19:41.453724+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15958,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419887+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:27.419887+0000","last_peered":"2024-09-15T03:19:27.419887+0000","last_clean":"2024-09-15T03:19:27.419887+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:27.419887+0000","last_undegraded":"2024-09-15T03:19:27.419887+0000","last_fullsized":"2024-09-15T03:19:27.419887+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16325,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419752+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:27.419752+0000","last_peered":"2024-09-15T03:19:27.419752+0000","last_clean":"2024-09-15T03:19:27.419752+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:27.419752+0000","last_undegraded":"2024-09-15T03:19:27.419752+0000","last_fullsized":"2024-09-15T03:19:27.419752+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.419754+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:27.419754+0000","last_peered":"2024-09-15T03:19:27.419754+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:27.419754+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12006,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.505878+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.505878+0000","last_peered":"2024-09-15T03:19:27.505878+0000","last_clean":"2024-09-15T03:19:27.505878+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.505878+0000","last_undegraded":"2024-09-15T03:19:27.505878+0000","last_fullsized":"2024-09-15T03:19:27.505878+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12140,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419978+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:27.419978+0000","last_peered":"2024-09-15T03:19:27.419978+0000","last_clean":"2024-09-15T03:19:27.419978+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:27.419978+0000","last_undegraded":"2024-09-15T03:19:27.419978+0000","last_fullsized":"2024-09-15T03:19:27.419978+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22786,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932168,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439672,"statfs":{"total":96636764160,"available":95682224128,"internally_reserved":0,"allocated":454656,"data_stored":196831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.504}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":450560,"data_stored":195881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.439},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63300000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.47399999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926084,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445756,"statfs":{"total":96636764160,"available":95688454144,"internally_reserved":0,"allocated":585728,"data_stored":332403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.80600000000000005}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919688,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452152,"statfs":{"total":96636764160,"available":95695003648,"internally_reserved":0,"allocated":454656,"data_stored":196831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.67400000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56799999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:45.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:45.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11995, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:27.423186+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:27.423186+0000', 'last_peered': '2024-09-15T03:19:27.423186+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:27.423186+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:45.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:45.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 12732, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:42.222901+0000', 'last_change': '2024-09-15T03:19:40.665587+0000', 'last_active': '2024-09-15T03:19:42.222901+0000', 'last_peered': '2024-09-15T03:19:42.222901+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:42.222901+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1900, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 204, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1899}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:45.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:45.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12101, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:27.423090+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:27.423090+0000', 'last_peered': '2024-09-15T03:19:27.423090+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:27.423090+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:45.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T03:19:45.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "37'11880", 'reported_seq': 16127, 'reported_epoch': 54, 'state': 'peering', 'last_fresh': '2024-09-15T03:19:41.453724+0000', 'last_change': '2024-09-15T03:19:41.457283+0000', 'last_active': '2024-09-15T03:19:41.453683+0000', 'last_peered': '2024-09-15T03:19:40.664154+0000', 'last_clean': '2024-09-15T03:05:05.354870+0000', 'last_became_active': '2024-09-15T03:06:09.250666+0000', 'last_became_peered': '2024-09-15T03:06:09.250666+0000', 'last_unstale': '2024-09-15T03:19:41.453724+0000', 'last_undegraded': '2024-09-15T03:19:41.453724+0000', 'last_fullsized': '2024-09-15T03:19:41.453724+0000', 'mapping_epoch': 54, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20907, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11877, 'num_write_kb': 47516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2084, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:45.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:45.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11799, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:27.419754+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:27.419754+0000', 'last_peered': '2024-09-15T03:19:27.419754+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:27.419754+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:45.109 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:45.487 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:45.487 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7919/112 objects degraded (7070.536%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":908,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3785908224,"bytes_avail":382761148416,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":7919,"degraded_total":112,"degraded_ratio":70.705357142857139,"recovering_objects_per_sec":153,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1226,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:45.504 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:45.504 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:45.824 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:45.827 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:45.837 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-15T03:19:45.028512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22786,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697176,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790184,"statfs":{"total":386547056640,"available":382761148416,"internally_reserved":0,"allocated":1945600,"data_stored":921946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002293"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11995,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.423186+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:27.423186+0000","last_peered":"2024-09-15T03:19:27.423186+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:27.423186+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14915,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.423070+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:27.423070+0000","last_peered":"2024-09-15T03:19:27.423070+0000","last_clean":"2024-09-15T03:19:27.423070+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:27.423070+0000","last_undegraded":"2024-09-15T03:19:27.423070+0000","last_fullsized":"2024-09-15T03:19:27.423070+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":12732,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:42.222901+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:42.222901+0000","last_peered":"2024-09-15T03:19:42.222901+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:42.222901+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1900,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1899}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15617,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419628+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:27.419628+0000","last_peered":"2024-09-15T03:19:27.419628+0000","last_clean":"2024-09-15T03:19:27.419628+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:27.419628+0000","last_undegraded":"2024-09-15T03:19:27.419628+0000","last_fullsized":"2024-09-15T03:19:27.419628+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16389,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419275+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:27.419275+0000","last_peered":"2024-09-15T03:19:27.419275+0000","last_clean":"2024-09-15T03:19:27.419275+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:27.419275+0000","last_undegraded":"2024-09-15T03:19:27.419275+0000","last_fullsized":"2024-09-15T03:19:27.419275+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12101,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.423090+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:27.423090+0000","last_peered":"2024-09-15T03:19:27.423090+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:27.423090+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16165,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.426280+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:27.426280+0000","last_peered":"2024-09-15T03:19:27.426280+0000","last_clean":"2024-09-15T03:19:27.426280+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:27.426280+0000","last_undegraded":"2024-09-15T03:19:27.426280+0000","last_fullsized":"2024-09-15T03:19:27.426280+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12228,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419604+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:27.419604+0000","last_peered":"2024-09-15T03:19:27.419604+0000","last_clean":"2024-09-15T03:19:27.419604+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:27.419604+0000","last_undegraded":"2024-09-15T03:19:27.419604+0000","last_fullsized":"2024-09-15T03:19:27.419604+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16149,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.422621+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:27.422621+0000","last_peered":"2024-09-15T03:19:27.422621+0000","last_clean":"2024-09-15T03:19:27.422621+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:27.422621+0000","last_undegraded":"2024-09-15T03:19:27.422621+0000","last_fullsized":"2024-09-15T03:19:27.422621+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16205,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.421132+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:27.421132+0000","last_peered":"2024-09-15T03:19:27.421132+0000","last_clean":"2024-09-15T03:19:27.421132+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:27.421132+0000","last_undegraded":"2024-09-15T03:19:27.421132+0000","last_fullsized":"2024-09-15T03:19:27.421132+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16127,"reported_epoch":54,"state":"peering","last_fresh":"2024-09-15T03:19:41.453724+0000","last_change":"2024-09-15T03:19:41.457283+0000","last_active":"2024-09-15T03:19:41.453683+0000","last_peered":"2024-09-15T03:19:40.664154+0000","last_clean":"2024-09-15T03:05:05.354870+0000","last_became_active":"2024-09-15T03:06:09.250666+0000","last_became_peered":"2024-09-15T03:06:09.250666+0000","last_unstale":"2024-09-15T03:19:41.453724+0000","last_undegraded":"2024-09-15T03:19:41.453724+0000","last_fullsized":"2024-09-15T03:19:41.453724+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15958,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419887+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:27.419887+0000","last_peered":"2024-09-15T03:19:27.419887+0000","last_clean":"2024-09-15T03:19:27.419887+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:27.419887+0000","last_undegraded":"2024-09-15T03:19:27.419887+0000","last_fullsized":"2024-09-15T03:19:27.419887+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16325,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419752+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:27.419752+0000","last_peered":"2024-09-15T03:19:27.419752+0000","last_clean":"2024-09-15T03:19:27.419752+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:27.419752+0000","last_undegraded":"2024-09-15T03:19:27.419752+0000","last_fullsized":"2024-09-15T03:19:27.419752+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:27.419754+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:27.419754+0000","last_peered":"2024-09-15T03:19:27.419754+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:27.419754+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12006,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.505878+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:27.505878+0000","last_peered":"2024-09-15T03:19:27.505878+0000","last_clean":"2024-09-15T03:19:27.505878+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:27.505878+0000","last_undegraded":"2024-09-15T03:19:27.505878+0000","last_fullsized":"2024-09-15T03:19:27.505878+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12140,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T03:19:27.419978+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:27.419978+0000","last_peered":"2024-09-15T03:19:27.419978+0000","last_clean":"2024-09-15T03:19:27.419978+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:27.419978+0000","last_undegraded":"2024-09-15T03:19:27.419978+0000","last_fullsized":"2024-09-15T03:19:27.419978+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22786,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932168,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439672,"statfs":{"total":96636764160,"available":95682224128,"internally_reserved":0,"allocated":454656,"data_stored":196831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.41499999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70199999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.504}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919236,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452604,"statfs":{"total":96636764160,"available":95695466496,"internally_reserved":0,"allocated":450560,"data_stored":195881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.439},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63300000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.47399999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926084,"kb_used_data":572,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445756,"statfs":{"total":96636764160,"available":95688454144,"internally_reserved":0,"allocated":585728,"data_stored":332403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.65300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.80600000000000005}]}]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919688,"kb_used_data":444,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452152,"statfs":{"total":96636764160,"available":95695003648,"internally_reserved":0,"allocated":454656,"data_stored":196831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.67400000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.56799999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:48.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:49.136 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:49.139 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:49.148 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-15T03:19:49.029577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23630,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705540,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781820,"statfs":{"total":386547056640,"available":382752583680,"internally_reserved":0,"allocated":1986560,"data_stored":940322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001925"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14916,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450830+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:41.450830+0000","last_peered":"2024-09-15T03:19:41.450830+0000","last_clean":"2024-09-15T03:19:41.450830+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:41.450830+0000","last_undegraded":"2024-09-15T03:19:41.450830+0000","last_fullsized":"2024-09-15T03:19:41.450830+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":14421,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:47.223121+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:47.223121+0000","last_peered":"2024-09-15T03:19:47.223121+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:47.223121+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15618,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450338+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:41.450338+0000","last_peered":"2024-09-15T03:19:41.450338+0000","last_clean":"2024-09-15T03:19:41.450338+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:41.450338+0000","last_undegraded":"2024-09-15T03:19:41.450338+0000","last_fullsized":"2024-09-15T03:19:41.450338+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12229,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450481+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:41.450481+0000","last_peered":"2024-09-15T03:19:41.450481+0000","last_clean":"2024-09-15T03:19:41.450481+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:41.450481+0000","last_undegraded":"2024-09-15T03:19:41.450481+0000","last_fullsized":"2024-09-15T03:19:41.450481+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15959,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450495+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:41.450495+0000","last_peered":"2024-09-15T03:19:41.450495+0000","last_clean":"2024-09-15T03:19:41.450495+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:41.450495+0000","last_undegraded":"2024-09-15T03:19:41.450495+0000","last_fullsized":"2024-09-15T03:19:41.450495+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16326,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450360+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:41.450360+0000","last_peered":"2024-09-15T03:19:41.450360+0000","last_clean":"2024-09-15T03:19:41.450360+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:41.450360+0000","last_undegraded":"2024-09-15T03:19:41.450360+0000","last_fullsized":"2024-09-15T03:19:41.450360+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12007,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450728+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:41.450728+0000","last_peered":"2024-09-15T03:19:41.450728+0000","last_clean":"2024-09-15T03:19:41.450728+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:41.450728+0000","last_undegraded":"2024-09-15T03:19:41.450728+0000","last_fullsized":"2024-09-15T03:19:41.450728+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12141,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450648+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:41.450648+0000","last_peered":"2024-09-15T03:19:41.450648+0000","last_clean":"2024-09-15T03:19:41.450648+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:41.450648+0000","last_undegraded":"2024-09-15T03:19:41.450648+0000","last_fullsized":"2024-09-15T03:19:41.450648+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23630,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.373},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.436},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.41999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.57899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.60999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934420,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437420,"statfs":{"total":96636764160,"available":95679918080,"internally_reserved":0,"allocated":602112,"data_stored":339766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.63900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.57899999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.36599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:49.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:49.447 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:49.450 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:49.462 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-15T03:19:49.029577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23630,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705540,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781820,"statfs":{"total":386547056640,"available":382752583680,"internally_reserved":0,"allocated":1986560,"data_stored":940322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001925"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14916,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450830+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:41.450830+0000","last_peered":"2024-09-15T03:19:41.450830+0000","last_clean":"2024-09-15T03:19:41.450830+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:41.450830+0000","last_undegraded":"2024-09-15T03:19:41.450830+0000","last_fullsized":"2024-09-15T03:19:41.450830+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":14421,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:47.223121+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:47.223121+0000","last_peered":"2024-09-15T03:19:47.223121+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:47.223121+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15618,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450338+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:41.450338+0000","last_peered":"2024-09-15T03:19:41.450338+0000","last_clean":"2024-09-15T03:19:41.450338+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:41.450338+0000","last_undegraded":"2024-09-15T03:19:41.450338+0000","last_fullsized":"2024-09-15T03:19:41.450338+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12229,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450481+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:41.450481+0000","last_peered":"2024-09-15T03:19:41.450481+0000","last_clean":"2024-09-15T03:19:41.450481+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:41.450481+0000","last_undegraded":"2024-09-15T03:19:41.450481+0000","last_fullsized":"2024-09-15T03:19:41.450481+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15959,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450495+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:41.450495+0000","last_peered":"2024-09-15T03:19:41.450495+0000","last_clean":"2024-09-15T03:19:41.450495+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:41.450495+0000","last_undegraded":"2024-09-15T03:19:41.450495+0000","last_fullsized":"2024-09-15T03:19:41.450495+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16326,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450360+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:41.450360+0000","last_peered":"2024-09-15T03:19:41.450360+0000","last_clean":"2024-09-15T03:19:41.450360+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:41.450360+0000","last_undegraded":"2024-09-15T03:19:41.450360+0000","last_fullsized":"2024-09-15T03:19:41.450360+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12007,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450728+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:41.450728+0000","last_peered":"2024-09-15T03:19:41.450728+0000","last_clean":"2024-09-15T03:19:41.450728+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:41.450728+0000","last_undegraded":"2024-09-15T03:19:41.450728+0000","last_fullsized":"2024-09-15T03:19:41.450728+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12141,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450648+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:41.450648+0000","last_peered":"2024-09-15T03:19:41.450648+0000","last_clean":"2024-09-15T03:19:41.450648+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:41.450648+0000","last_undegraded":"2024-09-15T03:19:41.450648+0000","last_fullsized":"2024-09-15T03:19:41.450648+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23630,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.373},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.436},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.41999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.57899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.60999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934420,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437420,"statfs":{"total":96636764160,"available":95679918080,"internally_reserved":0,"allocated":602112,"data_stored":339766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.63900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.57899999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.36599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:49.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11996, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.451523+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:41.451523+0000', 'last_peered': '2024-09-15T03:19:41.451523+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:41.451523+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 14421, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:47.223121+0000', 'last_change': '2024-09-15T03:19:40.665587+0000', 'last_active': '2024-09-15T03:19:47.223121+0000', 'last_peered': '2024-09-15T03:19:47.223121+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:47.223121+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1056, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1048, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1055}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12102, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.451508+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:41.451508+0000', 'last_peered': '2024-09-15T03:19:41.451508+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:41.451508+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:49.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11800, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.449914+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:41.449914+0000', 'last_peered': '2024-09-15T03:19:41.449914+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:41.449914+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:49.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:49.829 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:49.829 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7075/112 objects degraded (6316.964%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":912,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3794472960,"bytes_avail":382752583680,"bytes_total":386547056640,"degraded_objects":7075,"degraded_total":112,"degraded_ratio":63.169642857142854,"recovering_objects_per_sec":184,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:49.848 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:49.849 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:50.169 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:50.172 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:50.183 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-15T03:19:49.029577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23630,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705540,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781820,"statfs":{"total":386547056640,"available":382752583680,"internally_reserved":0,"allocated":1986560,"data_stored":940322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1476,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001925"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14916,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450830+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:41.450830+0000","last_peered":"2024-09-15T03:19:41.450830+0000","last_clean":"2024-09-15T03:19:41.450830+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:41.450830+0000","last_undegraded":"2024-09-15T03:19:41.450830+0000","last_fullsized":"2024-09-15T03:19:41.450830+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":14421,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:47.223121+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:47.223121+0000","last_peered":"2024-09-15T03:19:47.223121+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:47.223121+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1056,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1055}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15618,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450338+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:41.450338+0000","last_peered":"2024-09-15T03:19:41.450338+0000","last_clean":"2024-09-15T03:19:41.450338+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:41.450338+0000","last_undegraded":"2024-09-15T03:19:41.450338+0000","last_fullsized":"2024-09-15T03:19:41.450338+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12229,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450481+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:41.450481+0000","last_peered":"2024-09-15T03:19:41.450481+0000","last_clean":"2024-09-15T03:19:41.450481+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:41.450481+0000","last_undegraded":"2024-09-15T03:19:41.450481+0000","last_fullsized":"2024-09-15T03:19:41.450481+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15959,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450495+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:41.450495+0000","last_peered":"2024-09-15T03:19:41.450495+0000","last_clean":"2024-09-15T03:19:41.450495+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:41.450495+0000","last_undegraded":"2024-09-15T03:19:41.450495+0000","last_fullsized":"2024-09-15T03:19:41.450495+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16326,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450360+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:41.450360+0000","last_peered":"2024-09-15T03:19:41.450360+0000","last_clean":"2024-09-15T03:19:41.450360+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:41.450360+0000","last_undegraded":"2024-09-15T03:19:41.450360+0000","last_fullsized":"2024-09-15T03:19:41.450360+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12007,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450728+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:41.450728+0000","last_peered":"2024-09-15T03:19:41.450728+0000","last_clean":"2024-09-15T03:19:41.450728+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:41.450728+0000","last_undegraded":"2024-09-15T03:19:41.450728+0000","last_fullsized":"2024-09-15T03:19:41.450728+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12141,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450648+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:41.450648+0000","last_peered":"2024-09-15T03:19:41.450648+0000","last_clean":"2024-09-15T03:19:41.450648+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:41.450648+0000","last_undegraded":"2024-09-15T03:19:41.450648+0000","last_fullsized":"2024-09-15T03:19:41.450648+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7075,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23630,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.373},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.436},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.41999999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.57899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.60999999999999999}]}]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934420,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437420,"statfs":{"total":96636764160,"available":95679918080,"internally_reserved":0,"allocated":602112,"data_stored":339766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.39200000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.63900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.57899999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.36599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.41099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:53.185 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:53.515 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:53.519 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:53.531 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-15T03:19:53.030698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693060,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794300,"statfs":{"total":386547056640,"available":382765363200,"internally_reserved":0,"allocated":1986560,"data_stored":940322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1508,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.577159"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14916,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450830+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:41.450830+0000","last_peered":"2024-09-15T03:19:41.450830+0000","last_clean":"2024-09-15T03:19:41.450830+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:41.450830+0000","last_undegraded":"2024-09-15T03:19:41.450830+0000","last_fullsized":"2024-09-15T03:19:41.450830+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":15749,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:52.221538+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:52.221538+0000","last_peered":"2024-09-15T03:19:52.221538+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:52.221538+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15618,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450338+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:41.450338+0000","last_peered":"2024-09-15T03:19:41.450338+0000","last_clean":"2024-09-15T03:19:41.450338+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:41.450338+0000","last_undegraded":"2024-09-15T03:19:41.450338+0000","last_fullsized":"2024-09-15T03:19:41.450338+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12229,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450481+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:41.450481+0000","last_peered":"2024-09-15T03:19:41.450481+0000","last_clean":"2024-09-15T03:19:41.450481+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:41.450481+0000","last_undegraded":"2024-09-15T03:19:41.450481+0000","last_fullsized":"2024-09-15T03:19:41.450481+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15959,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450495+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:41.450495+0000","last_peered":"2024-09-15T03:19:41.450495+0000","last_clean":"2024-09-15T03:19:41.450495+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:41.450495+0000","last_undegraded":"2024-09-15T03:19:41.450495+0000","last_fullsized":"2024-09-15T03:19:41.450495+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16326,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450360+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:41.450360+0000","last_peered":"2024-09-15T03:19:41.450360+0000","last_clean":"2024-09-15T03:19:41.450360+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:41.450360+0000","last_undegraded":"2024-09-15T03:19:41.450360+0000","last_fullsized":"2024-09-15T03:19:41.450360+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12007,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450728+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:41.450728+0000","last_peered":"2024-09-15T03:19:41.450728+0000","last_clean":"2024-09-15T03:19:41.450728+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:41.450728+0000","last_undegraded":"2024-09-15T03:19:41.450728+0000","last_fullsized":"2024-09-15T03:19:41.450728+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12141,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450648+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:41.450648+0000","last_peered":"2024-09-15T03:19:41.450648+0000","last_clean":"2024-09-15T03:19:41.450648+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:41.450648+0000","last_undegraded":"2024-09-15T03:19:41.450648+0000","last_fullsized":"2024-09-15T03:19:41.450648+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.94199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.5}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921940,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449900,"statfs":{"total":96636764160,"available":95692697600,"internally_reserved":0,"allocated":602112,"data_stored":339766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.50600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.45700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.65700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:53.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:53.827 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:53.830 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:53.841 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-15T03:19:53.030698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693060,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794300,"statfs":{"total":386547056640,"available":382765363200,"internally_reserved":0,"allocated":1986560,"data_stored":940322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1508,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.577159"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14916,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450830+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:41.450830+0000","last_peered":"2024-09-15T03:19:41.450830+0000","last_clean":"2024-09-15T03:19:41.450830+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:41.450830+0000","last_undegraded":"2024-09-15T03:19:41.450830+0000","last_fullsized":"2024-09-15T03:19:41.450830+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":15749,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:52.221538+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:52.221538+0000","last_peered":"2024-09-15T03:19:52.221538+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:52.221538+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15618,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450338+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:41.450338+0000","last_peered":"2024-09-15T03:19:41.450338+0000","last_clean":"2024-09-15T03:19:41.450338+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:41.450338+0000","last_undegraded":"2024-09-15T03:19:41.450338+0000","last_fullsized":"2024-09-15T03:19:41.450338+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12229,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450481+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:41.450481+0000","last_peered":"2024-09-15T03:19:41.450481+0000","last_clean":"2024-09-15T03:19:41.450481+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:41.450481+0000","last_undegraded":"2024-09-15T03:19:41.450481+0000","last_fullsized":"2024-09-15T03:19:41.450481+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15959,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450495+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:41.450495+0000","last_peered":"2024-09-15T03:19:41.450495+0000","last_clean":"2024-09-15T03:19:41.450495+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:41.450495+0000","last_undegraded":"2024-09-15T03:19:41.450495+0000","last_fullsized":"2024-09-15T03:19:41.450495+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16326,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450360+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:41.450360+0000","last_peered":"2024-09-15T03:19:41.450360+0000","last_clean":"2024-09-15T03:19:41.450360+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:41.450360+0000","last_undegraded":"2024-09-15T03:19:41.450360+0000","last_fullsized":"2024-09-15T03:19:41.450360+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12007,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450728+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:41.450728+0000","last_peered":"2024-09-15T03:19:41.450728+0000","last_clean":"2024-09-15T03:19:41.450728+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:41.450728+0000","last_undegraded":"2024-09-15T03:19:41.450728+0000","last_fullsized":"2024-09-15T03:19:41.450728+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12141,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450648+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:41.450648+0000","last_peered":"2024-09-15T03:19:41.450648+0000","last_clean":"2024-09-15T03:19:41.450648+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:41.450648+0000","last_undegraded":"2024-09-15T03:19:41.450648+0000","last_fullsized":"2024-09-15T03:19:41.450648+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.94199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.5}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921940,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449900,"statfs":{"total":96636764160,"available":95692697600,"internally_reserved":0,"allocated":602112,"data_stored":339766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.50600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.45700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.65700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:53.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:53.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11996, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.451523+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:41.451523+0000', 'last_peered': '2024-09-15T03:19:41.451523+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:41.451523+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:53.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:53.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 15749, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:52.221538+0000', 'last_change': '2024-09-15T03:19:40.665587+0000', 'last_active': '2024-09-15T03:19:52.221538+0000', 'last_peered': '2024-09-15T03:19:52.221538+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:52.221538+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 392, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1712, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -391}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:53.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:53.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12102, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.451508+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:41.451508+0000', 'last_peered': '2024-09-15T03:19:41.451508+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:41.451508+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:53.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:53.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11800, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.449914+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:41.449914+0000', 'last_peered': '2024-09-15T03:19:41.449914+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:41.449914+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:53.843 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:54.194 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:54.194 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6411/112 objects degraded (5724.107%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":917,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3781693440,"bytes_avail":382765363200,"bytes_total":386547056640,"degraded_objects":6411,"degraded_total":112,"degraded_ratio":57.241071428571431,"recovering_objects_per_sec":142,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1508,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:54.212 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:54.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:54.518 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:54.521 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:54.532 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-15T03:19:53.030698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693060,"kb_used_data":1940,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794300,"statfs":{"total":386547056640,"available":382765363200,"internally_reserved":0,"allocated":1986560,"data_stored":940322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1508,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1508,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.577159"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14916,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450830+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:41.450830+0000","last_peered":"2024-09-15T03:19:41.450830+0000","last_clean":"2024-09-15T03:19:41.450830+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:41.450830+0000","last_undegraded":"2024-09-15T03:19:41.450830+0000","last_fullsized":"2024-09-15T03:19:41.450830+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":15749,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:52.221538+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:52.221538+0000","last_peered":"2024-09-15T03:19:52.221538+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:52.221538+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15618,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450338+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:41.450338+0000","last_peered":"2024-09-15T03:19:41.450338+0000","last_clean":"2024-09-15T03:19:41.450338+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:41.450338+0000","last_undegraded":"2024-09-15T03:19:41.450338+0000","last_fullsized":"2024-09-15T03:19:41.450338+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12229,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450481+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:41.450481+0000","last_peered":"2024-09-15T03:19:41.450481+0000","last_clean":"2024-09-15T03:19:41.450481+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:41.450481+0000","last_undegraded":"2024-09-15T03:19:41.450481+0000","last_fullsized":"2024-09-15T03:19:41.450481+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15959,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450495+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:41.450495+0000","last_peered":"2024-09-15T03:19:41.450495+0000","last_clean":"2024-09-15T03:19:41.450495+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:41.450495+0000","last_undegraded":"2024-09-15T03:19:41.450495+0000","last_fullsized":"2024-09-15T03:19:41.450495+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16326,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450360+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:41.450360+0000","last_peered":"2024-09-15T03:19:41.450360+0000","last_clean":"2024-09-15T03:19:41.450360+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:41.450360+0000","last_undegraded":"2024-09-15T03:19:41.450360+0000","last_fullsized":"2024-09-15T03:19:41.450360+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12007,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450728+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:41.450728+0000","last_peered":"2024-09-15T03:19:41.450728+0000","last_clean":"2024-09-15T03:19:41.450728+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:41.450728+0000","last_undegraded":"2024-09-15T03:19:41.450728+0000","last_fullsized":"2024-09-15T03:19:41.450728+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12141,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.450648+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:41.450648+0000","last_peered":"2024-09-15T03:19:41.450648+0000","last_clean":"2024-09-15T03:19:41.450648+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:41.450648+0000","last_undegraded":"2024-09-15T03:19:41.450648+0000","last_fullsized":"2024-09-15T03:19:41.450648+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.94199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.5}]}]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921940,"kb_used_data":588,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449900,"statfs":{"total":96636764160,"available":95692697600,"internally_reserved":0,"allocated":602112,"data_stored":339766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.50600000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.45700000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.65700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:57.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:57.850 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:57.853 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:57.863 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-15T03:19:57.032112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697232,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790128,"statfs":{"total":386547056640,"available":382761091072,"internally_reserved":0,"allocated":1994752,"data_stored":943977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002534"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":15749,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:52.221538+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:52.221538+0000","last_peered":"2024-09-15T03:19:52.221538+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:52.221538+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.94199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.5}]}]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926112,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445728,"statfs":{"total":96636764160,"available":95688425472,"internally_reserved":0,"allocated":610304,"data_stored":343421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.877}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.65700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:57.864 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:58.176 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:58.188 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-15T03:19:57.032112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697232,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790128,"statfs":{"total":386547056640,"available":382761091072,"internally_reserved":0,"allocated":1994752,"data_stored":943977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002534"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":15749,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:52.221538+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:52.221538+0000","last_peered":"2024-09-15T03:19:52.221538+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:52.221538+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.94199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.5}]}]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926112,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445728,"statfs":{"total":96636764160,"available":95688425472,"internally_reserved":0,"allocated":610304,"data_stored":343421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.877}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.65700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:19:58.189 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:19:58.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11996, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.451523+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:41.451523+0000', 'last_peered': '2024-09-15T03:19:41.451523+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:41.451523+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:58.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T03:19:58.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12220", 'reported_seq': 15749, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:52.221538+0000', 'last_change': '2024-09-15T03:19:40.665587+0000', 'last_active': '2024-09-15T03:19:52.221538+0000', 'last_peered': '2024-09-15T03:19:52.221538+0000', 'last_clean': '2024-09-15T03:05:05.206247+0000', 'last_became_active': '2024-09-15T03:06:09.288203+0000', 'last_became_peered': '2024-09-15T03:06:09.288203+0000', 'last_unstale': '2024-09-15T03:19:52.221538+0000', 'last_undegraded': '2024-09-15T03:06:08.913960+0000', 'last_fullsized': '2024-09-15T03:06:08.837597+0000', 'mapping_epoch': 30, 'log_start': "31'2200", 'ondisk_log_start': "31'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6969, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 392, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12219, 'num_write_kb': 48871, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1712, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -391}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T03:19:58.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:19:58.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12102, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.451508+0000', 'last_change': '2024-09-15T03:06:09.264079+0000', 'last_active': '2024-09-15T03:19:41.451508+0000', 'last_peered': '2024-09-15T03:19:41.451508+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:41.451508+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2038, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2038}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:58.190 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:19:58.190 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11800, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:41.449914+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:41.449914+0000', 'last_peered': '2024-09-15T03:19:41.449914+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:41.449914+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:19:58.190 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:19:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:58.564 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6411/112 objects degraded (5724.107%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":921,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3785965568,"bytes_avail":382761091072,"bytes_total":386547056640,"degraded_objects":6411,"degraded_total":112,"degraded_ratio":57.241071428571431,"recovering_objects_per_sec":82,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":664,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:19:58.577 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:19:58.578 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:19:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:19:58.884 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:19:58.894 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-15T03:19:57.032112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697232,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790128,"statfs":{"total":386547056640,"available":382761091072,"internally_reserved":0,"allocated":1994752,"data_stored":943977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-664,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002534"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11996,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451523+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:41.451523+0000","last_peered":"2024-09-15T03:19:41.451523+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:41.451523+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":15749,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:52.221538+0000","last_change":"2024-09-15T03:19:40.665587+0000","last_active":"2024-09-15T03:19:52.221538+0000","last_peered":"2024-09-15T03:19:52.221538+0000","last_clean":"2024-09-15T03:05:05.206247+0000","last_became_active":"2024-09-15T03:06:09.288203+0000","last_became_peered":"2024-09-15T03:06:09.288203+0000","last_unstale":"2024-09-15T03:19:52.221538+0000","last_undegraded":"2024-09-15T03:06:08.913960+0000","last_fullsized":"2024-09-15T03:06:08.837597+0000","mapping_epoch":30,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16390,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.449535+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:41.449535+0000","last_peered":"2024-09-15T03:19:41.449535+0000","last_clean":"2024-09-15T03:19:41.449535+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:41.449535+0000","last_undegraded":"2024-09-15T03:19:41.449535+0000","last_fullsized":"2024-09-15T03:19:41.449535+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12102,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.451508+0000","last_change":"2024-09-15T03:06:09.264079+0000","last_active":"2024-09-15T03:19:41.451508+0000","last_peered":"2024-09-15T03:19:41.451508+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:41.451508+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2038}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16166,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.459095+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:41.459095+0000","last_peered":"2024-09-15T03:19:41.459095+0000","last_clean":"2024-09-15T03:19:41.459095+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:41.459095+0000","last_undegraded":"2024-09-15T03:19:41.459095+0000","last_fullsized":"2024-09-15T03:19:41.459095+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16150,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.451300+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:41.451300+0000","last_peered":"2024-09-15T03:19:41.451300+0000","last_clean":"2024-09-15T03:19:41.451300+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:41.451300+0000","last_undegraded":"2024-09-15T03:19:41.451300+0000","last_fullsized":"2024-09-15T03:19:41.451300+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16206,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-15T03:19:41.453309+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:41.453309+0000","last_peered":"2024-09-15T03:19:41.453309+0000","last_clean":"2024-09-15T03:19:41.453309+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:41.453309+0000","last_undegraded":"2024-09-15T03:19:41.453309+0000","last_fullsized":"2024-09-15T03:19:41.453309+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16137,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T03:19:42.560096+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:42.560096+0000","last_peered":"2024-09-15T03:19:42.560096+0000","last_clean":"2024-09-15T03:19:42.560096+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:42.560096+0000","last_undegraded":"2024-09-15T03:19:42.560096+0000","last_fullsized":"2024-09-15T03:19:42.560096+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11800,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:41.449914+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:41.449914+0000","last_peered":"2024-09-15T03:19:41.449914+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:41.449914+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6411,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24294,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932176,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439664,"statfs":{"total":96636764160,"available":95682215936,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.641,"15min":1.6679999999999999},"min":{"1min":0.26000000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.54300000000000004,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.8680000000000001,"15min":1.968},"min":{"1min":0.316,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.84199999999999997,"5min":11.006,"15min":26.143000000000001},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":2.8050000000000002,"15min":2.5840000000000001},"min":{"1min":0.34999999999999998,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":2.4119999999999999,"5min":24.82,"15min":40.784999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57199999999999995,"5min":1.724,"15min":1.772},"min":{"1min":0.42499999999999999,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":0.76100000000000001,"5min":21.062000000000001,"15min":32.377000000000002},"last":0.94199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919248,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452592,"statfs":{"total":96636764160,"available":95695454208,"internally_reserved":0,"allocated":458752,"data_stored":199552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":2.4079999999999999,"15min":2.1930000000000001},"min":{"1min":0.25900000000000001,"5min":0.17399999999999999,"15min":0.16300000000000001},"max":{"1min":0.59099999999999997,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":1.2589999999999999,"15min":1.4590000000000001},"min":{"1min":0.28899999999999998,"5min":0.20000000000000001,"15min":0.192},"max":{"1min":0.86399999999999999,"5min":19.189,"15min":19.329000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:18:54 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.9279999999999999,"15min":1.8180000000000001},"min":{"1min":0.39500000000000002,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.79900000000000004,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.0070000000000001,"15min":1.9359999999999999},"min":{"1min":0.35999999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.84999999999999998,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.5}]}]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926112,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445728,"statfs":{"total":96636764160,"available":95688425472,"internally_reserved":0,"allocated":610304,"data_stored":343421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.877}]}]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919696,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452144,"statfs":{"total":96636764160,"available":95694995456,"internally_reserved":0,"allocated":462848,"data_stored":200502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48699999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":2.46,"15min":2.3540000000000001},"min":{"1min":0.32000000000000001,"5min":0.193,"15min":0.16200000000000001},"max":{"1min":0.81599999999999995,"5min":30.562999999999999,"15min":30.802},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.9019999999999999,"15min":1.732},"min":{"1min":0.29099999999999998,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.998,"5min":12.281000000000001,"15min":20.387},"last":0.65700000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.69699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:01.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:02.209 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:02.212 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:02.225 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-15T03:20:01.033205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25012,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697288,"kb_used_data":1996,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790072,"statfs":{"total":386547056640,"available":382761033728,"internally_reserved":0,"allocated":2043904,"data_stored":965844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002506"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12757,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:57.398030+0000","last_change":"2024-09-15T03:19:55.207349+0000","last_active":"2024-09-15T03:19:57.398030+0000","last_peered":"2024-09-15T03:19:57.398030+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:57.398030+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11802,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.501734+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:55.501734+0000","last_peered":"2024-09-15T03:19:55.501734+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:55.501734+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25012,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.437}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56599999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.52800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.54100000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.66200000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926112,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445728,"statfs":{"total":96636764160,"available":95688425472,"internally_reserved":0,"allocated":610304,"data_stored":343421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.877}]}]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:02.226 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:02.514 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:02.517 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:02.530 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-15T03:20:01.033205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25012,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697288,"kb_used_data":1996,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790072,"statfs":{"total":386547056640,"available":382761033728,"internally_reserved":0,"allocated":2043904,"data_stored":965844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002506"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":12757,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:57.398030+0000","last_change":"2024-09-15T03:19:55.207349+0000","last_active":"2024-09-15T03:19:57.398030+0000","last_peered":"2024-09-15T03:19:57.398030+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:19:57.398030+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":344,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11802,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.501734+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:55.501734+0000","last_peered":"2024-09-15T03:19:55.501734+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:55.501734+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25012,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.437}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.56599999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.52800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.54100000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.66200000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926112,"kb_used_data":596,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445728,"statfs":{"total":96636764160,"available":95688425472,"internally_reserved":0,"allocated":610304,"data_stored":343421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.877}]}]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":1.1120000000000001,"15min":1.399},"min":{"1min":0.38,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.89800000000000002,"5min":15.114000000000001,"15min":20.948},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":1.528,"15min":1.6040000000000001},"min":{"1min":0.435,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":1.361,"5min":16.562000000000001,"15min":30.013000000000002},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:02.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:20:02.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11998, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:55.504606+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:55.504606+0000', 'last_peered': '2024-09-15T03:19:55.504606+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:55.504606+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:02.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:20:02.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 12757, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:57.398030+0000', 'last_change': '2024-09-15T03:19:55.207349+0000', 'last_active': '2024-09-15T03:19:57.398030+0000', 'last_peered': '2024-09-15T03:19:57.398030+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:19:57.398030+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1712, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 344, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1712}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:02.531 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:20:02.531 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11802, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:55.501734+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:55.501734+0000', 'last_peered': '2024-09-15T03:19:55.501734+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:55.501734+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:02.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:20:02.906 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:02.906 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5693/112 objects degraded (5083.036%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":925,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3786022912,"bytes_avail":382761033728,"bytes_total":386547056640,"degraded_objects":5693,"degraded_total":112,"degraded_ratio":50.830357142857146,"recovering_objects_per_sec":89,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":718,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:20:02.920 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:20:02.920 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:03.234 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:03.236 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:03.246 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-09-15T03:20:03.033706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25858,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705612,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781748,"statfs":{"total":386547056640,"available":382752509952,"internally_reserved":0,"allocated":2052096,"data_stored":969478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002188"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":14449,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:02.396074+0000","last_change":"2024-09-15T03:19:55.207349+0000","last_active":"2024-09-15T03:20:02.396074+0000","last_peered":"2024-09-15T03:20:02.396074+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:20:02.396074+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11802,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.501734+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:55.501734+0000","last_peered":"2024-09-15T03:19:55.501734+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:55.501734+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25858,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.56799999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.81299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.72299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.377}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.71399999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.50800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934436,"kb_used_data":604,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437404,"statfs":{"total":96636764160,"available":95679901696,"internally_reserved":0,"allocated":618496,"data_stored":347055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.48399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.52300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.55000000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.65100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.80900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.70799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:06.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:06.566 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:06.569 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:06.580 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-15T03:20:05.034208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25858,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705612,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781748,"statfs":{"total":386547056640,"available":382752509952,"internally_reserved":0,"allocated":2052096,"data_stored":969478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.534706"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":14449,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:02.396074+0000","last_change":"2024-09-15T03:19:55.207349+0000","last_active":"2024-09-15T03:20:02.396074+0000","last_peered":"2024-09-15T03:20:02.396074+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:20:02.396074+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11802,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.501734+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:55.501734+0000","last_peered":"2024-09-15T03:19:55.501734+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:55.501734+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25858,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.56799999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.81299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.72299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.377}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.71399999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.50800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934436,"kb_used_data":604,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437404,"statfs":{"total":96636764160,"available":95679901696,"internally_reserved":0,"allocated":618496,"data_stored":347055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.48399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.52300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.55000000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.65100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.80900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.70799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:06.581 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:06.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:06.892 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:06.904 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-15T03:20:05.034208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25858,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705612,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781748,"statfs":{"total":386547056640,"available":382752509952,"internally_reserved":0,"allocated":2052096,"data_stored":969478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.534706"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":14449,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:02.396074+0000","last_change":"2024-09-15T03:19:55.207349+0000","last_active":"2024-09-15T03:20:02.396074+0000","last_peered":"2024-09-15T03:20:02.396074+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:20:02.396074+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11802,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.501734+0000","last_change":"2024-09-15T03:06:09.266146+0000","last_active":"2024-09-15T03:19:55.501734+0000","last_peered":"2024-09-15T03:19:55.501734+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:19:55.501734+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25858,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.56799999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.81299999999999994}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.72299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.377}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.71399999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.50800000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934436,"kb_used_data":604,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437404,"statfs":{"total":96636764160,"available":95679901696,"internally_reserved":0,"allocated":618496,"data_stored":347055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.48399999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.52300000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.55000000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247990,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.65100000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.80900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.70799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:06.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:20:06.905 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11998, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:55.504606+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:55.504606+0000', 'last_peered': '2024-09-15T03:19:55.504606+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:55.504606+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:06.905 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:20:06.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 14449, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:20:02.396074+0000', 'last_change': '2024-09-15T03:19:55.207349+0000', 'last_active': '2024-09-15T03:20:02.396074+0000', 'last_peered': '2024-09-15T03:20:02.396074+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:20:02.396074+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 866, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1190, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -866}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:06.906 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:20:06.906 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11802, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:55.501734+0000', 'last_change': '2024-09-15T03:06:09.266146+0000', 'last_active': '2024-09-15T03:19:55.501734+0000', 'last_peered': '2024-09-15T03:19:55.501734+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:19:55.501734+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2015, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:06.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:20:07.271 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:07.271 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4847/112 objects degraded (4327.679%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":930,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3794546688,"bytes_avail":382752509952,"bytes_total":386547056640,"degraded_objects":4847,"degraded_total":112,"degraded_ratio":43.276785714285715,"recovering_objects_per_sec":164,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1564,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:20:07.286 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:20:07.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:07.578 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:07.580 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:07.592 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-15T03:20:07.545686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26731,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697292,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790068,"statfs":{"total":386547056640,"available":382761029632,"internally_reserved":0,"allocated":2052096,"data_stored":969478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2437,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.513574"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16182,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:20:07.312018+0000","last_change":"2024-09-15T03:20:07.312557+0000","last_active":"2024-09-15T03:20:07.312018+0000","last_peered":"2024-09-15T03:20:07.312018+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:20:07.312018+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11817,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.398550+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:07.398550+0000","last_peered":"2024-09-15T03:20:07.398550+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:07.398550+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2006}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26731,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.502},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.434},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57699999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.496}]}]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926116,"kb_used_data":604,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445724,"statfs":{"total":96636764160,"available":95688421376,"internally_reserved":0,"allocated":618496,"data_stored":347055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.41799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.38400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.40400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.68600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:10.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:10.920 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:10.923 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:10.934 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-15T03:20:09.035420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26731,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697292,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790068,"statfs":{"total":386547056640,"available":382761029632,"internally_reserved":0,"allocated":2052096,"data_stored":969478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1719,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002214"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16182,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:20:07.312018+0000","last_change":"2024-09-15T03:20:07.312557+0000","last_active":"2024-09-15T03:20:07.312018+0000","last_peered":"2024-09-15T03:20:07.312018+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:20:07.312018+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11817,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.398550+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:07.398550+0000","last_peered":"2024-09-15T03:20:07.398550+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:07.398550+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2006}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26731,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.502},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.434},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57699999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.496}]}]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926116,"kb_used_data":604,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445724,"statfs":{"total":96636764160,"available":95688421376,"internally_reserved":0,"allocated":618496,"data_stored":347055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.41799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.38400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.40400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.68600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:10.935 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:11.268 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:11.271 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:11.281 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-15T03:20:11.035889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26731,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697292,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790068,"statfs":{"total":386547056640,"available":382761029632,"internally_reserved":0,"allocated":2052096,"data_stored":969478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002182"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16182,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:20:07.312018+0000","last_change":"2024-09-15T03:20:07.312557+0000","last_active":"2024-09-15T03:20:07.312018+0000","last_peered":"2024-09-15T03:20:07.312018+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:20:07.312018+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11817,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.398550+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:07.398550+0000","last_peered":"2024-09-15T03:20:07.398550+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:07.398550+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2006}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26731,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.502},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.434},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57699999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.496}]}]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926116,"kb_used_data":604,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445724,"statfs":{"total":96636764160,"available":95688421376,"internally_reserved":0,"allocated":618496,"data_stored":347055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.41799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.38400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.40400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.68600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:11.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:20:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 11998, 'reported_epoch': 56, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:19:55.504606+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:19:55.504606+0000', 'last_peered': '2024-09-15T03:19:55.504606+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:19:55.504606+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T03:20:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11998", 'reported_seq': 16182, 'reported_epoch': 57, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-15T03:20:07.312018+0000', 'last_change': '2024-09-15T03:20:07.312557+0000', 'last_active': '2024-09-15T03:20:07.312018+0000', 'last_peered': '2024-09-15T03:20:07.312018+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.250548+0000', 'last_became_peered': '2024-09-15T03:06:09.250548+0000', 'last_unstale': '2024-09-15T03:20:07.312018+0000', 'last_undegraded': '2024-09-15T03:06:08.899142+0000', 'last_fullsized': '2024-09-15T03:06:08.835736+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11998, 'num_write_kb': 47992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2056, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '1(3)', '2(2)'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:20:11.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 11817, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:20:07.398550+0000', 'last_change': '2024-09-15T03:20:07.314010+0000', 'last_active': '2024-09-15T03:20:07.398550+0000', 'last_peered': '2024-09-15T03:20:07.398550+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:20:07.398550+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2008, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2006}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:11.283 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:20:11.659 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:11.659 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3974/112 objects degraded (3548.214%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":934,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3786027008,"bytes_avail":382761029632,"bytes_total":386547056640,"degraded_objects":3974,"degraded_total":112,"degraded_ratio":35.482142857142854,"recovering_objects_per_sec":109,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:20:11.673 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:20:11.673 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:11.992 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:11.994 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:12.007 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-15T03:20:11.035889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26731,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697292,"kb_used_data":2004,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373790068,"statfs":{"total":386547056640,"available":382761029632,"internally_reserved":0,"allocated":2052096,"data_stored":969478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-873,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002182"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":11998,"reported_epoch":56,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:19:55.504606+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:19:55.504606+0000","last_peered":"2024-09-15T03:19:55.504606+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:19:55.504606+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14918,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504301+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:19:55.504301+0000","last_peered":"2024-09-15T03:19:55.504301+0000","last_clean":"2024-09-15T03:19:55.504301+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:19:55.504301+0000","last_undegraded":"2024-09-15T03:19:55.504301+0000","last_fullsized":"2024-09-15T03:19:55.504301+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16547,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T03:19:56.602224+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:19:56.602224+0000","last_peered":"2024-09-15T03:19:56.602224+0000","last_clean":"2024-09-15T03:19:56.602224+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:19:56.602224+0000","last_undegraded":"2024-09-15T03:19:56.602224+0000","last_fullsized":"2024-09-15T03:19:56.602224+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15620,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503307+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:19:55.503307+0000","last_peered":"2024-09-15T03:19:55.503307+0000","last_clean":"2024-09-15T03:19:55.503307+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:19:55.503307+0000","last_undegraded":"2024-09-15T03:19:55.503307+0000","last_fullsized":"2024-09-15T03:19:55.503307+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16392,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.500844+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:19:55.500844+0000","last_peered":"2024-09-15T03:19:55.500844+0000","last_clean":"2024-09-15T03:19:55.500844+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:19:55.500844+0000","last_undegraded":"2024-09-15T03:19:55.500844+0000","last_fullsized":"2024-09-15T03:19:55.500844+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16182,"reported_epoch":57,"state":"active+undersized+remapped","last_fresh":"2024-09-15T03:20:07.312018+0000","last_change":"2024-09-15T03:20:07.312557+0000","last_active":"2024-09-15T03:20:07.312018+0000","last_peered":"2024-09-15T03:20:07.312018+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.250548+0000","last_became_peered":"2024-09-15T03:06:09.250548+0000","last_unstale":"2024-09-15T03:20:07.312018+0000","last_undegraded":"2024-09-15T03:06:08.899142+0000","last_fullsized":"2024-09-15T03:06:08.835736+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:05:44.155436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","1(3)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16168,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.511842+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:19:55.511842+0000","last_peered":"2024-09-15T03:19:55.511842+0000","last_clean":"2024-09-15T03:19:55.511842+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:19:55.511842+0000","last_undegraded":"2024-09-15T03:19:55.511842+0000","last_fullsized":"2024-09-15T03:19:55.511842+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12231,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503631+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:19:55.503631+0000","last_peered":"2024-09-15T03:19:55.503631+0000","last_clean":"2024-09-15T03:19:55.503631+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:19:55.503631+0000","last_undegraded":"2024-09-15T03:19:55.503631+0000","last_fullsized":"2024-09-15T03:19:55.503631+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.501120+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:19:55.501120+0000","last_peered":"2024-09-15T03:19:55.501120+0000","last_clean":"2024-09-15T03:19:55.501120+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:19:55.501120+0000","last_undegraded":"2024-09-15T03:19:55.501120+0000","last_fullsized":"2024-09-15T03:19:55.501120+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16208,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503201+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:19:55.503201+0000","last_peered":"2024-09-15T03:19:55.503201+0000","last_clean":"2024-09-15T03:19:55.503201+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:19:55.503201+0000","last_undegraded":"2024-09-15T03:19:55.503201+0000","last_fullsized":"2024-09-15T03:19:55.503201+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503983+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:19:55.503983+0000","last_peered":"2024-09-15T03:19:55.503983+0000","last_clean":"2024-09-15T03:19:55.503983+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:19:55.503983+0000","last_undegraded":"2024-09-15T03:19:55.503983+0000","last_fullsized":"2024-09-15T03:19:55.503983+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15961,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.504172+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:19:55.504172+0000","last_peered":"2024-09-15T03:19:55.504172+0000","last_clean":"2024-09-15T03:19:55.504172+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:19:55.504172+0000","last_undegraded":"2024-09-15T03:19:55.504172+0000","last_fullsized":"2024-09-15T03:19:55.504172+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16328,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503454+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:19:55.503454+0000","last_peered":"2024-09-15T03:19:55.503454+0000","last_clean":"2024-09-15T03:19:55.503454+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:19:55.503454+0000","last_undegraded":"2024-09-15T03:19:55.503454+0000","last_fullsized":"2024-09-15T03:19:55.503454+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11817,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.398550+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:07.398550+0000","last_peered":"2024-09-15T03:20:07.398550+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:07.398550+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2006}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12009,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503602+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:19:55.503602+0000","last_peered":"2024-09-15T03:19:55.503602+0000","last_clean":"2024-09-15T03:19:55.503602+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:19:55.503602+0000","last_undegraded":"2024-09-15T03:19:55.503602+0000","last_fullsized":"2024-09-15T03:19:55.503602+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12143,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-15T03:19:55.503779+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:19:55.503779+0000","last_peered":"2024-09-15T03:19:55.503779+0000","last_clean":"2024-09-15T03:19:55.503779+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:19:55.503779+0000","last_undegraded":"2024-09-15T03:19:55.503779+0000","last_fullsized":"2024-09-15T03:19:55.503779+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3974,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26731,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":155648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932196,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439644,"statfs":{"total":96636764160,"available":95682195456,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.502},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919264,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452576,"statfs":{"total":96636764160,"available":95695437824,"internally_reserved":0,"allocated":475136,"data_stored":206841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.434},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57699999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.63200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.496}]}]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926116,"kb_used_data":604,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445724,"statfs":{"total":96636764160,"available":95688421376,"internally_reserved":0,"allocated":618496,"data_stored":347055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.41799999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.38400000000000001}]}]},{"osd":0,"up_from":23,"seq":98784247991,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919716,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452124,"statfs":{"total":96636764160,"available":95694974976,"internally_reserved":0,"allocated":479232,"data_stored":207791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.40400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.68600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":143360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:15.009 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:15.333 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:15.335 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:15.348 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-15T03:20:15.037156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27551,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701512,"kb_used_data":2060,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785848,"statfs":{"total":386547056640,"available":382756708352,"internally_reserved":0,"allocated":2109440,"data_stored":990146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002408"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12000,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.548419+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:20:07.548419+0000","last_peered":"2024-09-15T03:20:07.548419+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:07.548419+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14920,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552884+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:07.552884+0000","last_peered":"2024-09-15T03:20:07.552884+0000","last_clean":"2024-09-15T03:20:07.552884+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:07.552884+0000","last_undegraded":"2024-09-15T03:20:07.552884+0000","last_fullsized":"2024-09-15T03:20:07.552884+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16549,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552817+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:07.552817+0000","last_peered":"2024-09-15T03:20:07.552817+0000","last_clean":"2024-09-15T03:20:07.552817+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:07.552817+0000","last_undegraded":"2024-09-15T03:20:07.552817+0000","last_fullsized":"2024-09-15T03:20:07.552817+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15622,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552725+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:07.552725+0000","last_peered":"2024-09-15T03:20:07.552725+0000","last_clean":"2024-09-15T03:20:07.552725+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:07.552725+0000","last_undegraded":"2024-09-15T03:20:07.552725+0000","last_fullsized":"2024-09-15T03:20:07.552725+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16394,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547346+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:07.547346+0000","last_peered":"2024-09-15T03:20:07.547346+0000","last_clean":"2024-09-15T03:20:07.547346+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:07.547346+0000","last_undegraded":"2024-09-15T03:20:07.547346+0000","last_fullsized":"2024-09-15T03:20:07.547346+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16195,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T03:20:08.644599+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:08.644599+0000","last_peered":"2024-09-15T03:20:08.644599+0000","last_clean":"2024-09-15T03:20:08.644599+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:08.644599+0000","last_undegraded":"2024-09-15T03:20:08.644599+0000","last_fullsized":"2024-09-15T03:20:08.644599+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16170,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.556550+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:07.556550+0000","last_peered":"2024-09-15T03:20:07.556550+0000","last_clean":"2024-09-15T03:20:07.556550+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:07.556550+0000","last_undegraded":"2024-09-15T03:20:07.556550+0000","last_fullsized":"2024-09-15T03:20:07.556550+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12233,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552641+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:07.552641+0000","last_peered":"2024-09-15T03:20:07.552641+0000","last_clean":"2024-09-15T03:20:07.552641+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:07.552641+0000","last_undegraded":"2024-09-15T03:20:07.552641+0000","last_fullsized":"2024-09-15T03:20:07.552641+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16154,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547384+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:07.547384+0000","last_peered":"2024-09-15T03:20:07.547384+0000","last_clean":"2024-09-15T03:20:07.547384+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:07.547384+0000","last_undegraded":"2024-09-15T03:20:07.547384+0000","last_fullsized":"2024-09-15T03:20:07.547384+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16210,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552919+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:07.552919+0000","last_peered":"2024-09-15T03:20:07.552919+0000","last_clean":"2024-09-15T03:20:07.552919+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:07.552919+0000","last_undegraded":"2024-09-15T03:20:07.552919+0000","last_fullsized":"2024-09-15T03:20:07.552919+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16141,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547658+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:07.547658+0000","last_peered":"2024-09-15T03:20:07.547658+0000","last_clean":"2024-09-15T03:20:07.547658+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:07.547658+0000","last_undegraded":"2024-09-15T03:20:07.547658+0000","last_fullsized":"2024-09-15T03:20:07.547658+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15963,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552998+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:07.552998+0000","last_peered":"2024-09-15T03:20:07.552998+0000","last_clean":"2024-09-15T03:20:07.552998+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:07.552998+0000","last_undegraded":"2024-09-15T03:20:07.552998+0000","last_fullsized":"2024-09-15T03:20:07.552998+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16330,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.553117+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:07.553117+0000","last_peered":"2024-09-15T03:20:07.553117+0000","last_clean":"2024-09-15T03:20:07.553117+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:07.553117+0000","last_undegraded":"2024-09-15T03:20:07.553117+0000","last_fullsized":"2024-09-15T03:20:07.553117+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":13459,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:12.403738+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:12.403738+0000","last_peered":"2024-09-15T03:20:12.403738+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:12.403738+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552926+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:07.552926+0000","last_peered":"2024-09-15T03:20:07.552926+0000","last_clean":"2024-09-15T03:20:07.552926+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:07.552926+0000","last_undegraded":"2024-09-15T03:20:07.552926+0000","last_fullsized":"2024-09-15T03:20:07.552926+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12145,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552752+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:07.552752+0000","last_peered":"2024-09-15T03:20:07.552752+0000","last_clean":"2024-09-15T03:20:07.552752+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:07.552752+0000","last_undegraded":"2024-09-15T03:20:07.552752+0000","last_fullsized":"2024-09-15T03:20:07.552752+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27551,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932212,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439628,"statfs":{"total":96636764160,"available":95682179072,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.57199999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.85699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":491520,"data_stored":214056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.53500000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930288,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441552,"statfs":{"total":96636764160,"available":95684149248,"internally_reserved":0,"allocated":626688,"data_stored":346078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.436},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.70099999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919732,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452108,"statfs":{"total":96636764160,"available":95694958592,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48099999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.39500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:15.349 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:15.642 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:15.645 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:15.655 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-15T03:20:15.037156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27551,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701512,"kb_used_data":2060,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785848,"statfs":{"total":386547056640,"available":382756708352,"internally_reserved":0,"allocated":2109440,"data_stored":990146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002408"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12000,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.548419+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:20:07.548419+0000","last_peered":"2024-09-15T03:20:07.548419+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:07.548419+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14920,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552884+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:07.552884+0000","last_peered":"2024-09-15T03:20:07.552884+0000","last_clean":"2024-09-15T03:20:07.552884+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:07.552884+0000","last_undegraded":"2024-09-15T03:20:07.552884+0000","last_fullsized":"2024-09-15T03:20:07.552884+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16549,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552817+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:07.552817+0000","last_peered":"2024-09-15T03:20:07.552817+0000","last_clean":"2024-09-15T03:20:07.552817+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:07.552817+0000","last_undegraded":"2024-09-15T03:20:07.552817+0000","last_fullsized":"2024-09-15T03:20:07.552817+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15622,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552725+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:07.552725+0000","last_peered":"2024-09-15T03:20:07.552725+0000","last_clean":"2024-09-15T03:20:07.552725+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:07.552725+0000","last_undegraded":"2024-09-15T03:20:07.552725+0000","last_fullsized":"2024-09-15T03:20:07.552725+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16394,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547346+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:07.547346+0000","last_peered":"2024-09-15T03:20:07.547346+0000","last_clean":"2024-09-15T03:20:07.547346+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:07.547346+0000","last_undegraded":"2024-09-15T03:20:07.547346+0000","last_fullsized":"2024-09-15T03:20:07.547346+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16195,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T03:20:08.644599+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:08.644599+0000","last_peered":"2024-09-15T03:20:08.644599+0000","last_clean":"2024-09-15T03:20:08.644599+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:08.644599+0000","last_undegraded":"2024-09-15T03:20:08.644599+0000","last_fullsized":"2024-09-15T03:20:08.644599+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16170,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.556550+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:07.556550+0000","last_peered":"2024-09-15T03:20:07.556550+0000","last_clean":"2024-09-15T03:20:07.556550+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:07.556550+0000","last_undegraded":"2024-09-15T03:20:07.556550+0000","last_fullsized":"2024-09-15T03:20:07.556550+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12233,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552641+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:07.552641+0000","last_peered":"2024-09-15T03:20:07.552641+0000","last_clean":"2024-09-15T03:20:07.552641+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:07.552641+0000","last_undegraded":"2024-09-15T03:20:07.552641+0000","last_fullsized":"2024-09-15T03:20:07.552641+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16154,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547384+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:07.547384+0000","last_peered":"2024-09-15T03:20:07.547384+0000","last_clean":"2024-09-15T03:20:07.547384+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:07.547384+0000","last_undegraded":"2024-09-15T03:20:07.547384+0000","last_fullsized":"2024-09-15T03:20:07.547384+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16210,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552919+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:07.552919+0000","last_peered":"2024-09-15T03:20:07.552919+0000","last_clean":"2024-09-15T03:20:07.552919+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:07.552919+0000","last_undegraded":"2024-09-15T03:20:07.552919+0000","last_fullsized":"2024-09-15T03:20:07.552919+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16141,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547658+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:07.547658+0000","last_peered":"2024-09-15T03:20:07.547658+0000","last_clean":"2024-09-15T03:20:07.547658+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:07.547658+0000","last_undegraded":"2024-09-15T03:20:07.547658+0000","last_fullsized":"2024-09-15T03:20:07.547658+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15963,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552998+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:07.552998+0000","last_peered":"2024-09-15T03:20:07.552998+0000","last_clean":"2024-09-15T03:20:07.552998+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:07.552998+0000","last_undegraded":"2024-09-15T03:20:07.552998+0000","last_fullsized":"2024-09-15T03:20:07.552998+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16330,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.553117+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:07.553117+0000","last_peered":"2024-09-15T03:20:07.553117+0000","last_clean":"2024-09-15T03:20:07.553117+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:07.553117+0000","last_undegraded":"2024-09-15T03:20:07.553117+0000","last_fullsized":"2024-09-15T03:20:07.553117+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":13459,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:12.403738+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:12.403738+0000","last_peered":"2024-09-15T03:20:12.403738+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:12.403738+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552926+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:07.552926+0000","last_peered":"2024-09-15T03:20:07.552926+0000","last_clean":"2024-09-15T03:20:07.552926+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:07.552926+0000","last_undegraded":"2024-09-15T03:20:07.552926+0000","last_fullsized":"2024-09-15T03:20:07.552926+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12145,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552752+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:07.552752+0000","last_peered":"2024-09-15T03:20:07.552752+0000","last_clean":"2024-09-15T03:20:07.552752+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:07.552752+0000","last_undegraded":"2024-09-15T03:20:07.552752+0000","last_fullsized":"2024-09-15T03:20:07.552752+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27551,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932212,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439628,"statfs":{"total":96636764160,"available":95682179072,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.57199999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.85699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":491520,"data_stored":214056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.53500000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930288,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441552,"statfs":{"total":96636764160,"available":95684149248,"internally_reserved":0,"allocated":626688,"data_stored":346078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.436},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.70099999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919732,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452108,"statfs":{"total":96636764160,"available":95694958592,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48099999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.39500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:15.656 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:20:15.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 12000, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:20:07.548419+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:20:07.548419+0000', 'last_peered': '2024-09-15T03:20:07.548419+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:20:07.548419+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:15.657 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:20:15.657 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 13459, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:20:12.403738+0000', 'last_change': '2024-09-15T03:20:07.314010+0000', 'last_active': '2024-09-15T03:20:12.403738+0000', 'last_peered': '2024-09-15T03:20:12.403738+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:20:12.403738+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1188, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 852, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:15.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:20:16.030 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:16.030 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3154/112 objects degraded (2816.071%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":939,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3790348288,"bytes_avail":382756708352,"bytes_total":386547056640,"degraded_objects":3154,"degraded_total":112,"degraded_ratio":28.160714285714285,"recovering_objects_per_sec":211,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:20:16.043 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:20:16.043 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:16.341 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:16.343 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:16.353 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-09-15T03:20:15.037156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27551,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701512,"kb_used_data":2060,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785848,"statfs":{"total":386547056640,"available":382756708352,"internally_reserved":0,"allocated":2109440,"data_stored":990146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1693,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002408"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12000,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.548419+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:20:07.548419+0000","last_peered":"2024-09-15T03:20:07.548419+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:07.548419+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14920,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552884+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:07.552884+0000","last_peered":"2024-09-15T03:20:07.552884+0000","last_clean":"2024-09-15T03:20:07.552884+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:07.552884+0000","last_undegraded":"2024-09-15T03:20:07.552884+0000","last_fullsized":"2024-09-15T03:20:07.552884+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16549,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552817+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:07.552817+0000","last_peered":"2024-09-15T03:20:07.552817+0000","last_clean":"2024-09-15T03:20:07.552817+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:07.552817+0000","last_undegraded":"2024-09-15T03:20:07.552817+0000","last_fullsized":"2024-09-15T03:20:07.552817+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15622,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552725+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:07.552725+0000","last_peered":"2024-09-15T03:20:07.552725+0000","last_clean":"2024-09-15T03:20:07.552725+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:07.552725+0000","last_undegraded":"2024-09-15T03:20:07.552725+0000","last_fullsized":"2024-09-15T03:20:07.552725+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16394,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547346+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:07.547346+0000","last_peered":"2024-09-15T03:20:07.547346+0000","last_clean":"2024-09-15T03:20:07.547346+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:07.547346+0000","last_undegraded":"2024-09-15T03:20:07.547346+0000","last_fullsized":"2024-09-15T03:20:07.547346+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16195,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T03:20:08.644599+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:08.644599+0000","last_peered":"2024-09-15T03:20:08.644599+0000","last_clean":"2024-09-15T03:20:08.644599+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:08.644599+0000","last_undegraded":"2024-09-15T03:20:08.644599+0000","last_fullsized":"2024-09-15T03:20:08.644599+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16170,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.556550+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:07.556550+0000","last_peered":"2024-09-15T03:20:07.556550+0000","last_clean":"2024-09-15T03:20:07.556550+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:07.556550+0000","last_undegraded":"2024-09-15T03:20:07.556550+0000","last_fullsized":"2024-09-15T03:20:07.556550+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12233,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552641+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:07.552641+0000","last_peered":"2024-09-15T03:20:07.552641+0000","last_clean":"2024-09-15T03:20:07.552641+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:07.552641+0000","last_undegraded":"2024-09-15T03:20:07.552641+0000","last_fullsized":"2024-09-15T03:20:07.552641+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16154,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547384+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:07.547384+0000","last_peered":"2024-09-15T03:20:07.547384+0000","last_clean":"2024-09-15T03:20:07.547384+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:07.547384+0000","last_undegraded":"2024-09-15T03:20:07.547384+0000","last_fullsized":"2024-09-15T03:20:07.547384+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16210,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552919+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:07.552919+0000","last_peered":"2024-09-15T03:20:07.552919+0000","last_clean":"2024-09-15T03:20:07.552919+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:07.552919+0000","last_undegraded":"2024-09-15T03:20:07.552919+0000","last_fullsized":"2024-09-15T03:20:07.552919+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16141,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547658+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:07.547658+0000","last_peered":"2024-09-15T03:20:07.547658+0000","last_clean":"2024-09-15T03:20:07.547658+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:07.547658+0000","last_undegraded":"2024-09-15T03:20:07.547658+0000","last_fullsized":"2024-09-15T03:20:07.547658+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15963,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552998+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:07.552998+0000","last_peered":"2024-09-15T03:20:07.552998+0000","last_clean":"2024-09-15T03:20:07.552998+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:07.552998+0000","last_undegraded":"2024-09-15T03:20:07.552998+0000","last_fullsized":"2024-09-15T03:20:07.552998+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16330,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.553117+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:07.553117+0000","last_peered":"2024-09-15T03:20:07.553117+0000","last_clean":"2024-09-15T03:20:07.553117+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:07.553117+0000","last_undegraded":"2024-09-15T03:20:07.553117+0000","last_fullsized":"2024-09-15T03:20:07.553117+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":13459,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:12.403738+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:12.403738+0000","last_peered":"2024-09-15T03:20:12.403738+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:12.403738+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552926+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:07.552926+0000","last_peered":"2024-09-15T03:20:07.552926+0000","last_clean":"2024-09-15T03:20:07.552926+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:07.552926+0000","last_undegraded":"2024-09-15T03:20:07.552926+0000","last_fullsized":"2024-09-15T03:20:07.552926+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12145,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552752+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:07.552752+0000","last_peered":"2024-09-15T03:20:07.552752+0000","last_clean":"2024-09-15T03:20:07.552752+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:07.552752+0000","last_undegraded":"2024-09-15T03:20:07.552752+0000","last_fullsized":"2024-09-15T03:20:07.552752+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27551,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932212,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439628,"statfs":{"total":96636764160,"available":95682179072,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.57199999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.85699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":491520,"data_stored":214056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.44700000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.53500000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930288,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441552,"statfs":{"total":96636764160,"available":95684149248,"internally_reserved":0,"allocated":626688,"data_stored":346078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.436},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.379}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.70099999999999996}]}]},{"osd":0,"up_from":23,"seq":98784247992,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919732,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452108,"statfs":{"total":96636764160,"available":95694958592,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48099999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.39500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:19.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:19.674 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:19.676 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:19.693 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-15T03:20:19.124478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28438,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693188,"kb_used_data":2060,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794172,"statfs":{"total":386547056640,"available":382765232128,"internally_reserved":0,"allocated":2109440,"data_stored":990146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1707,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.089057"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12000,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.548419+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:20:07.548419+0000","last_peered":"2024-09-15T03:20:07.548419+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:07.548419+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14920,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552884+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:07.552884+0000","last_peered":"2024-09-15T03:20:07.552884+0000","last_clean":"2024-09-15T03:20:07.552884+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:07.552884+0000","last_undegraded":"2024-09-15T03:20:07.552884+0000","last_fullsized":"2024-09-15T03:20:07.552884+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16549,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552817+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:07.552817+0000","last_peered":"2024-09-15T03:20:07.552817+0000","last_clean":"2024-09-15T03:20:07.552817+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:07.552817+0000","last_undegraded":"2024-09-15T03:20:07.552817+0000","last_fullsized":"2024-09-15T03:20:07.552817+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15622,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552725+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:07.552725+0000","last_peered":"2024-09-15T03:20:07.552725+0000","last_clean":"2024-09-15T03:20:07.552725+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:07.552725+0000","last_undegraded":"2024-09-15T03:20:07.552725+0000","last_fullsized":"2024-09-15T03:20:07.552725+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16394,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547346+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:07.547346+0000","last_peered":"2024-09-15T03:20:07.547346+0000","last_clean":"2024-09-15T03:20:07.547346+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:07.547346+0000","last_undegraded":"2024-09-15T03:20:07.547346+0000","last_fullsized":"2024-09-15T03:20:07.547346+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16195,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T03:20:08.644599+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:08.644599+0000","last_peered":"2024-09-15T03:20:08.644599+0000","last_clean":"2024-09-15T03:20:08.644599+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:08.644599+0000","last_undegraded":"2024-09-15T03:20:08.644599+0000","last_fullsized":"2024-09-15T03:20:08.644599+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16170,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.556550+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:07.556550+0000","last_peered":"2024-09-15T03:20:07.556550+0000","last_clean":"2024-09-15T03:20:07.556550+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:07.556550+0000","last_undegraded":"2024-09-15T03:20:07.556550+0000","last_fullsized":"2024-09-15T03:20:07.556550+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12233,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552641+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:07.552641+0000","last_peered":"2024-09-15T03:20:07.552641+0000","last_clean":"2024-09-15T03:20:07.552641+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:07.552641+0000","last_undegraded":"2024-09-15T03:20:07.552641+0000","last_fullsized":"2024-09-15T03:20:07.552641+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16154,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547384+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:07.547384+0000","last_peered":"2024-09-15T03:20:07.547384+0000","last_clean":"2024-09-15T03:20:07.547384+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:07.547384+0000","last_undegraded":"2024-09-15T03:20:07.547384+0000","last_fullsized":"2024-09-15T03:20:07.547384+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16210,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552919+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:07.552919+0000","last_peered":"2024-09-15T03:20:07.552919+0000","last_clean":"2024-09-15T03:20:07.552919+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:07.552919+0000","last_undegraded":"2024-09-15T03:20:07.552919+0000","last_fullsized":"2024-09-15T03:20:07.552919+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16141,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547658+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:07.547658+0000","last_peered":"2024-09-15T03:20:07.547658+0000","last_clean":"2024-09-15T03:20:07.547658+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:07.547658+0000","last_undegraded":"2024-09-15T03:20:07.547658+0000","last_fullsized":"2024-09-15T03:20:07.547658+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15963,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552998+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:07.552998+0000","last_peered":"2024-09-15T03:20:07.552998+0000","last_clean":"2024-09-15T03:20:07.552998+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:07.552998+0000","last_undegraded":"2024-09-15T03:20:07.552998+0000","last_fullsized":"2024-09-15T03:20:07.552998+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16330,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.553117+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:07.553117+0000","last_peered":"2024-09-15T03:20:07.553117+0000","last_clean":"2024-09-15T03:20:07.553117+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:07.553117+0000","last_undegraded":"2024-09-15T03:20:07.553117+0000","last_fullsized":"2024-09-15T03:20:07.553117+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":15233,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:17.405288+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:17.405288+0000","last_peered":"2024-09-15T03:20:17.405288+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:17.405288+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1739,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552926+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:07.552926+0000","last_peered":"2024-09-15T03:20:07.552926+0000","last_clean":"2024-09-15T03:20:07.552926+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:07.552926+0000","last_undegraded":"2024-09-15T03:20:07.552926+0000","last_fullsized":"2024-09-15T03:20:07.552926+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12145,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552752+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:07.552752+0000","last_peered":"2024-09-15T03:20:07.552752+0000","last_clean":"2024-09-15T03:20:07.552752+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:07.552752+0000","last_undegraded":"2024-09-15T03:20:07.552752+0000","last_fullsized":"2024-09-15T03:20:07.552752+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28438,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932212,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439628,"statfs":{"total":96636764160,"available":95682179072,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.46400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.5},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.63}]}]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":491520,"data_stored":214056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.30599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.35199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921964,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449876,"statfs":{"total":96636764160,"available":95692673024,"internally_reserved":0,"allocated":626688,"data_stored":346078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.60999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.91100000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919732,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452108,"statfs":{"total":96636764160,"available":95694958592,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.61899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.64500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.434},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:19.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:19.996 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:19.998 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:20.010 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-15T03:20:19.124478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28438,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693188,"kb_used_data":2060,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794172,"statfs":{"total":386547056640,"available":382765232128,"internally_reserved":0,"allocated":2109440,"data_stored":990146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1707,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.089057"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12000,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.548419+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:20:07.548419+0000","last_peered":"2024-09-15T03:20:07.548419+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:07.548419+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14920,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552884+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:07.552884+0000","last_peered":"2024-09-15T03:20:07.552884+0000","last_clean":"2024-09-15T03:20:07.552884+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:07.552884+0000","last_undegraded":"2024-09-15T03:20:07.552884+0000","last_fullsized":"2024-09-15T03:20:07.552884+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16549,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552817+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:07.552817+0000","last_peered":"2024-09-15T03:20:07.552817+0000","last_clean":"2024-09-15T03:20:07.552817+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:07.552817+0000","last_undegraded":"2024-09-15T03:20:07.552817+0000","last_fullsized":"2024-09-15T03:20:07.552817+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15622,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552725+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:07.552725+0000","last_peered":"2024-09-15T03:20:07.552725+0000","last_clean":"2024-09-15T03:20:07.552725+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:07.552725+0000","last_undegraded":"2024-09-15T03:20:07.552725+0000","last_fullsized":"2024-09-15T03:20:07.552725+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16394,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547346+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:07.547346+0000","last_peered":"2024-09-15T03:20:07.547346+0000","last_clean":"2024-09-15T03:20:07.547346+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:07.547346+0000","last_undegraded":"2024-09-15T03:20:07.547346+0000","last_fullsized":"2024-09-15T03:20:07.547346+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16195,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T03:20:08.644599+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:08.644599+0000","last_peered":"2024-09-15T03:20:08.644599+0000","last_clean":"2024-09-15T03:20:08.644599+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:08.644599+0000","last_undegraded":"2024-09-15T03:20:08.644599+0000","last_fullsized":"2024-09-15T03:20:08.644599+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16170,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.556550+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:07.556550+0000","last_peered":"2024-09-15T03:20:07.556550+0000","last_clean":"2024-09-15T03:20:07.556550+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:07.556550+0000","last_undegraded":"2024-09-15T03:20:07.556550+0000","last_fullsized":"2024-09-15T03:20:07.556550+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12233,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552641+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:07.552641+0000","last_peered":"2024-09-15T03:20:07.552641+0000","last_clean":"2024-09-15T03:20:07.552641+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:07.552641+0000","last_undegraded":"2024-09-15T03:20:07.552641+0000","last_fullsized":"2024-09-15T03:20:07.552641+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16154,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547384+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:07.547384+0000","last_peered":"2024-09-15T03:20:07.547384+0000","last_clean":"2024-09-15T03:20:07.547384+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:07.547384+0000","last_undegraded":"2024-09-15T03:20:07.547384+0000","last_fullsized":"2024-09-15T03:20:07.547384+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16210,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552919+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:07.552919+0000","last_peered":"2024-09-15T03:20:07.552919+0000","last_clean":"2024-09-15T03:20:07.552919+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:07.552919+0000","last_undegraded":"2024-09-15T03:20:07.552919+0000","last_fullsized":"2024-09-15T03:20:07.552919+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16141,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547658+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:07.547658+0000","last_peered":"2024-09-15T03:20:07.547658+0000","last_clean":"2024-09-15T03:20:07.547658+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:07.547658+0000","last_undegraded":"2024-09-15T03:20:07.547658+0000","last_fullsized":"2024-09-15T03:20:07.547658+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15963,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552998+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:07.552998+0000","last_peered":"2024-09-15T03:20:07.552998+0000","last_clean":"2024-09-15T03:20:07.552998+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:07.552998+0000","last_undegraded":"2024-09-15T03:20:07.552998+0000","last_fullsized":"2024-09-15T03:20:07.552998+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16330,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.553117+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:07.553117+0000","last_peered":"2024-09-15T03:20:07.553117+0000","last_clean":"2024-09-15T03:20:07.553117+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:07.553117+0000","last_undegraded":"2024-09-15T03:20:07.553117+0000","last_fullsized":"2024-09-15T03:20:07.553117+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":15233,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:17.405288+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:17.405288+0000","last_peered":"2024-09-15T03:20:17.405288+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:17.405288+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1739,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552926+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:07.552926+0000","last_peered":"2024-09-15T03:20:07.552926+0000","last_clean":"2024-09-15T03:20:07.552926+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:07.552926+0000","last_undegraded":"2024-09-15T03:20:07.552926+0000","last_fullsized":"2024-09-15T03:20:07.552926+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12145,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552752+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:07.552752+0000","last_peered":"2024-09-15T03:20:07.552752+0000","last_clean":"2024-09-15T03:20:07.552752+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:07.552752+0000","last_undegraded":"2024-09-15T03:20:07.552752+0000","last_fullsized":"2024-09-15T03:20:07.552752+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28438,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932212,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439628,"statfs":{"total":96636764160,"available":95682179072,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.46400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.5},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.63}]}]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":491520,"data_stored":214056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.30599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.35199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921964,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449876,"statfs":{"total":96636764160,"available":95692673024,"internally_reserved":0,"allocated":626688,"data_stored":346078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.60999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.91100000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919732,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452108,"statfs":{"total":96636764160,"available":95694958592,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.61899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.64500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.434},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:20.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:20:20.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 12000, 'reported_epoch': 58, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:20:07.548419+0000', 'last_change': '2024-09-15T03:06:09.295046+0000', 'last_active': '2024-09-15T03:20:07.548419+0000', 'last_peered': '2024-09-15T03:20:07.548419+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:20:07.548419+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1966, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1966}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:20.012 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T03:20:20.012 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11680", 'reported_seq': 15233, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:20:17.405288+0000', 'last_change': '2024-09-15T03:20:07.314010+0000', 'last_active': '2024-09-15T03:20:17.405288+0000', 'last_peered': '2024-09-15T03:20:17.405288+0000', 'last_clean': '2024-09-15T03:05:04.808490+0000', 'last_became_active': '2024-09-15T03:06:09.254329+0000', 'last_became_peered': '2024-09-15T03:06:09.254329+0000', 'last_unstale': '2024-09-15T03:20:17.405288+0000', 'last_undegraded': '2024-09-15T03:06:08.914738+0000', 'last_fullsized': '2024-09-15T03:06:08.838058+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13938, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11678, 'num_write_kb': 46724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1739, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:20.013 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:20:20.448 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:20.448 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2267/112 objects degraded (2024.107%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":943,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3781824512,"bytes_avail":382765232128,"bytes_total":386547056640,"degraded_objects":2267,"degraded_total":112,"degraded_ratio":20.241071428571427,"recovering_objects_per_sec":162,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1707,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:20:20.465 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T03:20:20.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:20.774 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:20.776 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:20.786 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-15T03:20:20.125312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28438,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3693188,"kb_used_data":2060,"kb_used_omap":14,"kb_used_meta":3691057,"kb_avail":373794172,"statfs":{"total":386547056640,"available":382765232128,"internally_reserved":0,"allocated":2109440,"data_stored":990146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3779642395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1707,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.089423"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12000,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:07.548419+0000","last_change":"2024-09-15T03:06:09.295046+0000","last_active":"2024-09-15T03:20:07.548419+0000","last_peered":"2024-09-15T03:20:07.548419+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:07.548419+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1966}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14920,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552884+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:07.552884+0000","last_peered":"2024-09-15T03:20:07.552884+0000","last_clean":"2024-09-15T03:20:07.552884+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:07.552884+0000","last_undegraded":"2024-09-15T03:20:07.552884+0000","last_fullsized":"2024-09-15T03:20:07.552884+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16549,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552817+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:07.552817+0000","last_peered":"2024-09-15T03:20:07.552817+0000","last_clean":"2024-09-15T03:20:07.552817+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:07.552817+0000","last_undegraded":"2024-09-15T03:20:07.552817+0000","last_fullsized":"2024-09-15T03:20:07.552817+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15622,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552725+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:07.552725+0000","last_peered":"2024-09-15T03:20:07.552725+0000","last_clean":"2024-09-15T03:20:07.552725+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:07.552725+0000","last_undegraded":"2024-09-15T03:20:07.552725+0000","last_fullsized":"2024-09-15T03:20:07.552725+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16394,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547346+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:07.547346+0000","last_peered":"2024-09-15T03:20:07.547346+0000","last_clean":"2024-09-15T03:20:07.547346+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:07.547346+0000","last_undegraded":"2024-09-15T03:20:07.547346+0000","last_fullsized":"2024-09-15T03:20:07.547346+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16195,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T03:20:08.644599+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:08.644599+0000","last_peered":"2024-09-15T03:20:08.644599+0000","last_clean":"2024-09-15T03:20:08.644599+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:08.644599+0000","last_undegraded":"2024-09-15T03:20:08.644599+0000","last_fullsized":"2024-09-15T03:20:08.644599+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16170,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.556550+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:07.556550+0000","last_peered":"2024-09-15T03:20:07.556550+0000","last_clean":"2024-09-15T03:20:07.556550+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:07.556550+0000","last_undegraded":"2024-09-15T03:20:07.556550+0000","last_fullsized":"2024-09-15T03:20:07.556550+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12233,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552641+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:07.552641+0000","last_peered":"2024-09-15T03:20:07.552641+0000","last_clean":"2024-09-15T03:20:07.552641+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:07.552641+0000","last_undegraded":"2024-09-15T03:20:07.552641+0000","last_fullsized":"2024-09-15T03:20:07.552641+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16154,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547384+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:07.547384+0000","last_peered":"2024-09-15T03:20:07.547384+0000","last_clean":"2024-09-15T03:20:07.547384+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:07.547384+0000","last_undegraded":"2024-09-15T03:20:07.547384+0000","last_fullsized":"2024-09-15T03:20:07.547384+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16210,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552919+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:07.552919+0000","last_peered":"2024-09-15T03:20:07.552919+0000","last_clean":"2024-09-15T03:20:07.552919+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:07.552919+0000","last_undegraded":"2024-09-15T03:20:07.552919+0000","last_fullsized":"2024-09-15T03:20:07.552919+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16141,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.547658+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:07.547658+0000","last_peered":"2024-09-15T03:20:07.547658+0000","last_clean":"2024-09-15T03:20:07.547658+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:07.547658+0000","last_undegraded":"2024-09-15T03:20:07.547658+0000","last_fullsized":"2024-09-15T03:20:07.547658+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15963,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552998+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:07.552998+0000","last_peered":"2024-09-15T03:20:07.552998+0000","last_clean":"2024-09-15T03:20:07.552998+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:07.552998+0000","last_undegraded":"2024-09-15T03:20:07.552998+0000","last_fullsized":"2024-09-15T03:20:07.552998+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16330,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.553117+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:07.553117+0000","last_peered":"2024-09-15T03:20:07.553117+0000","last_clean":"2024-09-15T03:20:07.553117+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:07.553117+0000","last_undegraded":"2024-09-15T03:20:07.553117+0000","last_fullsized":"2024-09-15T03:20:07.553117+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":15233,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:17.405288+0000","last_change":"2024-09-15T03:20:07.314010+0000","last_active":"2024-09-15T03:20:17.405288+0000","last_peered":"2024-09-15T03:20:17.405288+0000","last_clean":"2024-09-15T03:05:04.808490+0000","last_became_active":"2024-09-15T03:06:09.254329+0000","last_became_peered":"2024-09-15T03:06:09.254329+0000","last_unstale":"2024-09-15T03:20:17.405288+0000","last_undegraded":"2024-09-15T03:06:08.914738+0000","last_fullsized":"2024-09-15T03:06:08.838058+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1739,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12011,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552926+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:07.552926+0000","last_peered":"2024-09-15T03:20:07.552926+0000","last_clean":"2024-09-15T03:20:07.552926+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:07.552926+0000","last_undegraded":"2024-09-15T03:20:07.552926+0000","last_fullsized":"2024-09-15T03:20:07.552926+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12145,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-15T03:20:07.552752+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:07.552752+0000","last_peered":"2024-09-15T03:20:07.552752+0000","last_clean":"2024-09-15T03:20:07.552752+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:07.552752+0000","last_undegraded":"2024-09-15T03:20:07.552752+0000","last_fullsized":"2024-09-15T03:20:07.552752+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28438,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932212,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439628,"statfs":{"total":96636764160,"available":95682179072,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.46400000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.5},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.63}]}]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919280,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452560,"statfs":{"total":96636764160,"available":95695421440,"internally_reserved":0,"allocated":491520,"data_stored":214056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.30599999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.35199999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":921964,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":921339,"kb_avail":93449876,"statfs":{"total":96636764160,"available":95692673024,"internally_reserved":0,"allocated":626688,"data_stored":346078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":943451713},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.60999999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.91100000000000003}]}]},{"osd":0,"up_from":23,"seq":98784247993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919732,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452108,"statfs":{"total":96636764160,"available":95694958592,"internally_reserved":0,"allocated":495616,"data_stored":215006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.61899999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.64500000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.434},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:23.789 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:24.113 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:24.115 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:24.126 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-15T03:20:23.039426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701568,"kb_used_data":2124,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785792,"statfs":{"total":386547056640,"available":382756651008,"internally_reserved":0,"allocated":2174976,"data_stored":1018710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-359,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002270"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12939,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:22.405844+0000","last_change":"2024-09-15T03:20:19.046425+0000","last_active":"2024-09-15T03:20:22.405844+0000","last_peered":"2024-09-15T03:20:22.405844+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:22.405844+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14922,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129322+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:19.129322+0000","last_peered":"2024-09-15T03:20:19.129322+0000","last_clean":"2024-09-15T03:20:19.129322+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:19.129322+0000","last_undegraded":"2024-09-15T03:20:19.129322+0000","last_fullsized":"2024-09-15T03:20:19.129322+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127558+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:19.127558+0000","last_peered":"2024-09-15T03:20:19.127558+0000","last_clean":"2024-09-15T03:20:19.127558+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:19.127558+0000","last_undegraded":"2024-09-15T03:20:19.127558+0000","last_fullsized":"2024-09-15T03:20:19.127558+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15624,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128698+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:19.128698+0000","last_peered":"2024-09-15T03:20:19.128698+0000","last_clean":"2024-09-15T03:20:19.128698+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:19.128698+0000","last_undegraded":"2024-09-15T03:20:19.128698+0000","last_fullsized":"2024-09-15T03:20:19.128698+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16396,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125977+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:19.125977+0000","last_peered":"2024-09-15T03:20:19.125977+0000","last_clean":"2024-09-15T03:20:19.125977+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:19.125977+0000","last_undegraded":"2024-09-15T03:20:19.125977+0000","last_fullsized":"2024-09-15T03:20:19.125977+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16197,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127174+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:19.127174+0000","last_peered":"2024-09-15T03:20:19.127174+0000","last_clean":"2024-09-15T03:20:19.127174+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:19.127174+0000","last_undegraded":"2024-09-15T03:20:19.127174+0000","last_fullsized":"2024-09-15T03:20:19.127174+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16172,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.132113+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:19.132113+0000","last_peered":"2024-09-15T03:20:19.132113+0000","last_clean":"2024-09-15T03:20:19.132113+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:19.132113+0000","last_undegraded":"2024-09-15T03:20:19.132113+0000","last_fullsized":"2024-09-15T03:20:19.132113+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12235,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128900+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:19.128900+0000","last_peered":"2024-09-15T03:20:19.128900+0000","last_clean":"2024-09-15T03:20:19.128900+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:19.128900+0000","last_undegraded":"2024-09-15T03:20:19.128900+0000","last_fullsized":"2024-09-15T03:20:19.128900+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16156,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125999+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:19.125999+0000","last_peered":"2024-09-15T03:20:19.125999+0000","last_clean":"2024-09-15T03:20:19.125999+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:19.125999+0000","last_undegraded":"2024-09-15T03:20:19.125999+0000","last_fullsized":"2024-09-15T03:20:19.125999+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16212,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127708+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:19.127708+0000","last_peered":"2024-09-15T03:20:19.127708+0000","last_clean":"2024-09-15T03:20:19.127708+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:19.127708+0000","last_undegraded":"2024-09-15T03:20:19.127708+0000","last_fullsized":"2024-09-15T03:20:19.127708+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16143,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127273+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:19.127273+0000","last_peered":"2024-09-15T03:20:19.127273+0000","last_clean":"2024-09-15T03:20:19.127273+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:19.127273+0000","last_undegraded":"2024-09-15T03:20:19.127273+0000","last_fullsized":"2024-09-15T03:20:19.127273+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15965,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129140+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:19.129140+0000","last_peered":"2024-09-15T03:20:19.129140+0000","last_clean":"2024-09-15T03:20:19.129140+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:19.129140+0000","last_undegraded":"2024-09-15T03:20:19.129140+0000","last_fullsized":"2024-09-15T03:20:19.129140+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16332,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128740+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:19.128740+0000","last_peered":"2024-09-15T03:20:19.128740+0000","last_clean":"2024-09-15T03:20:19.128740+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:19.128740+0000","last_undegraded":"2024-09-15T03:20:19.128740+0000","last_fullsized":"2024-09-15T03:20:19.128740+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267276+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267276+0000","last_peered":"2024-09-15T03:20:20.267276+0000","last_clean":"2024-09-15T03:20:20.267276+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267276+0000","last_undegraded":"2024-09-15T03:20:20.267276+0000","last_fullsized":"2024-09-15T03:20:20.267276+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12013,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128929+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:19.128929+0000","last_peered":"2024-09-15T03:20:19.128929+0000","last_clean":"2024-09-15T03:20:19.128929+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:19.128929+0000","last_undegraded":"2024-09-15T03:20:19.128929+0000","last_fullsized":"2024-09-15T03:20:19.128929+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12147,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129049+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:19.129049+0000","last_peered":"2024-09-15T03:20:19.129049+0000","last_clean":"2024-09-15T03:20:19.129049+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:19.129049+0000","last_undegraded":"2024-09-15T03:20:19.129049+0000","last_fullsized":"2024-09-15T03:20:19.129049+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932228,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439612,"statfs":{"total":96636764160,"available":95682162688,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.497},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.67300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919292,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452548,"statfs":{"total":96636764160,"available":95695409152,"internally_reserved":0,"allocated":507904,"data_stored":221197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.439}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930300,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441540,"statfs":{"total":96636764160,"available":95684136960,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.58499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.54800000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919748,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452092,"statfs":{"total":96636764160,"available":95694942208,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:24.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:24.438 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:24.442 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:24.454 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-15T03:20:23.039426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701568,"kb_used_data":2124,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785792,"statfs":{"total":386547056640,"available":382756651008,"internally_reserved":0,"allocated":2174976,"data_stored":1018710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-359,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002270"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12939,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:22.405844+0000","last_change":"2024-09-15T03:20:19.046425+0000","last_active":"2024-09-15T03:20:22.405844+0000","last_peered":"2024-09-15T03:20:22.405844+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:22.405844+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14922,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129322+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:19.129322+0000","last_peered":"2024-09-15T03:20:19.129322+0000","last_clean":"2024-09-15T03:20:19.129322+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:19.129322+0000","last_undegraded":"2024-09-15T03:20:19.129322+0000","last_fullsized":"2024-09-15T03:20:19.129322+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127558+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:19.127558+0000","last_peered":"2024-09-15T03:20:19.127558+0000","last_clean":"2024-09-15T03:20:19.127558+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:19.127558+0000","last_undegraded":"2024-09-15T03:20:19.127558+0000","last_fullsized":"2024-09-15T03:20:19.127558+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15624,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128698+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:19.128698+0000","last_peered":"2024-09-15T03:20:19.128698+0000","last_clean":"2024-09-15T03:20:19.128698+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:19.128698+0000","last_undegraded":"2024-09-15T03:20:19.128698+0000","last_fullsized":"2024-09-15T03:20:19.128698+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16396,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125977+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:19.125977+0000","last_peered":"2024-09-15T03:20:19.125977+0000","last_clean":"2024-09-15T03:20:19.125977+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:19.125977+0000","last_undegraded":"2024-09-15T03:20:19.125977+0000","last_fullsized":"2024-09-15T03:20:19.125977+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16197,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127174+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:19.127174+0000","last_peered":"2024-09-15T03:20:19.127174+0000","last_clean":"2024-09-15T03:20:19.127174+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:19.127174+0000","last_undegraded":"2024-09-15T03:20:19.127174+0000","last_fullsized":"2024-09-15T03:20:19.127174+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16172,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.132113+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:19.132113+0000","last_peered":"2024-09-15T03:20:19.132113+0000","last_clean":"2024-09-15T03:20:19.132113+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:19.132113+0000","last_undegraded":"2024-09-15T03:20:19.132113+0000","last_fullsized":"2024-09-15T03:20:19.132113+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12235,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128900+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:19.128900+0000","last_peered":"2024-09-15T03:20:19.128900+0000","last_clean":"2024-09-15T03:20:19.128900+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:19.128900+0000","last_undegraded":"2024-09-15T03:20:19.128900+0000","last_fullsized":"2024-09-15T03:20:19.128900+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16156,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125999+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:19.125999+0000","last_peered":"2024-09-15T03:20:19.125999+0000","last_clean":"2024-09-15T03:20:19.125999+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:19.125999+0000","last_undegraded":"2024-09-15T03:20:19.125999+0000","last_fullsized":"2024-09-15T03:20:19.125999+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16212,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127708+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:19.127708+0000","last_peered":"2024-09-15T03:20:19.127708+0000","last_clean":"2024-09-15T03:20:19.127708+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:19.127708+0000","last_undegraded":"2024-09-15T03:20:19.127708+0000","last_fullsized":"2024-09-15T03:20:19.127708+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16143,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127273+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:19.127273+0000","last_peered":"2024-09-15T03:20:19.127273+0000","last_clean":"2024-09-15T03:20:19.127273+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:19.127273+0000","last_undegraded":"2024-09-15T03:20:19.127273+0000","last_fullsized":"2024-09-15T03:20:19.127273+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15965,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129140+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:19.129140+0000","last_peered":"2024-09-15T03:20:19.129140+0000","last_clean":"2024-09-15T03:20:19.129140+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:19.129140+0000","last_undegraded":"2024-09-15T03:20:19.129140+0000","last_fullsized":"2024-09-15T03:20:19.129140+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16332,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128740+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:19.128740+0000","last_peered":"2024-09-15T03:20:19.128740+0000","last_clean":"2024-09-15T03:20:19.128740+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:19.128740+0000","last_undegraded":"2024-09-15T03:20:19.128740+0000","last_fullsized":"2024-09-15T03:20:19.128740+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267276+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267276+0000","last_peered":"2024-09-15T03:20:20.267276+0000","last_clean":"2024-09-15T03:20:20.267276+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267276+0000","last_undegraded":"2024-09-15T03:20:20.267276+0000","last_fullsized":"2024-09-15T03:20:20.267276+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12013,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128929+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:19.128929+0000","last_peered":"2024-09-15T03:20:19.128929+0000","last_clean":"2024-09-15T03:20:19.128929+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:19.128929+0000","last_undegraded":"2024-09-15T03:20:19.128929+0000","last_fullsized":"2024-09-15T03:20:19.128929+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12147,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129049+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:19.129049+0000","last_peered":"2024-09-15T03:20:19.129049+0000","last_clean":"2024-09-15T03:20:19.129049+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:19.129049+0000","last_undegraded":"2024-09-15T03:20:19.129049+0000","last_fullsized":"2024-09-15T03:20:19.129049+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932228,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439612,"statfs":{"total":96636764160,"available":95682162688,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.497},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.67300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919292,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452548,"statfs":{"total":96636764160,"available":95695409152,"internally_reserved":0,"allocated":507904,"data_stored":221197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.439}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930300,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441540,"statfs":{"total":96636764160,"available":95684136960,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.58499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.54800000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919748,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452092,"statfs":{"total":96636764160,"available":95694942208,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:24.455 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:20:24.456 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 12939, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:20:22.405844+0000', 'last_change': '2024-09-15T03:20:19.046425+0000', 'last_active': '2024-09-15T03:20:22.405844+0000', 'last_peered': '2024-09-15T03:20:22.405844+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:20:22.405844+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1498, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 478, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1498}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:24.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:20:24.827 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:24.828 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1498/112 objects degraded (1337.500%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":947,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3790405632,"bytes_avail":382756651008,"bytes_total":386547056640,"degraded_objects":1498,"degraded_total":112,"degraded_ratio":13.375},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:20:24.847 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T03:20:24.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:25.142 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:25.145 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:25.161 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2024-09-15T03:20:25.040000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701568,"kb_used_data":2124,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785792,"statfs":{"total":386547056640,"available":382756651008,"internally_reserved":0,"allocated":2174976,"data_stored":1018710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-359,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002355"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12939,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:22.405844+0000","last_change":"2024-09-15T03:20:19.046425+0000","last_active":"2024-09-15T03:20:22.405844+0000","last_peered":"2024-09-15T03:20:22.405844+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:22.405844+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14922,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129322+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:19.129322+0000","last_peered":"2024-09-15T03:20:19.129322+0000","last_clean":"2024-09-15T03:20:19.129322+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:19.129322+0000","last_undegraded":"2024-09-15T03:20:19.129322+0000","last_fullsized":"2024-09-15T03:20:19.129322+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127558+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:19.127558+0000","last_peered":"2024-09-15T03:20:19.127558+0000","last_clean":"2024-09-15T03:20:19.127558+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:19.127558+0000","last_undegraded":"2024-09-15T03:20:19.127558+0000","last_fullsized":"2024-09-15T03:20:19.127558+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15624,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128698+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:19.128698+0000","last_peered":"2024-09-15T03:20:19.128698+0000","last_clean":"2024-09-15T03:20:19.128698+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:19.128698+0000","last_undegraded":"2024-09-15T03:20:19.128698+0000","last_fullsized":"2024-09-15T03:20:19.128698+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16396,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125977+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:19.125977+0000","last_peered":"2024-09-15T03:20:19.125977+0000","last_clean":"2024-09-15T03:20:19.125977+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:19.125977+0000","last_undegraded":"2024-09-15T03:20:19.125977+0000","last_fullsized":"2024-09-15T03:20:19.125977+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16197,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127174+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:19.127174+0000","last_peered":"2024-09-15T03:20:19.127174+0000","last_clean":"2024-09-15T03:20:19.127174+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:19.127174+0000","last_undegraded":"2024-09-15T03:20:19.127174+0000","last_fullsized":"2024-09-15T03:20:19.127174+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16172,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.132113+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:19.132113+0000","last_peered":"2024-09-15T03:20:19.132113+0000","last_clean":"2024-09-15T03:20:19.132113+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:19.132113+0000","last_undegraded":"2024-09-15T03:20:19.132113+0000","last_fullsized":"2024-09-15T03:20:19.132113+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12235,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128900+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:19.128900+0000","last_peered":"2024-09-15T03:20:19.128900+0000","last_clean":"2024-09-15T03:20:19.128900+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:19.128900+0000","last_undegraded":"2024-09-15T03:20:19.128900+0000","last_fullsized":"2024-09-15T03:20:19.128900+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16156,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125999+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:19.125999+0000","last_peered":"2024-09-15T03:20:19.125999+0000","last_clean":"2024-09-15T03:20:19.125999+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:19.125999+0000","last_undegraded":"2024-09-15T03:20:19.125999+0000","last_fullsized":"2024-09-15T03:20:19.125999+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16212,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127708+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:19.127708+0000","last_peered":"2024-09-15T03:20:19.127708+0000","last_clean":"2024-09-15T03:20:19.127708+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:19.127708+0000","last_undegraded":"2024-09-15T03:20:19.127708+0000","last_fullsized":"2024-09-15T03:20:19.127708+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16143,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127273+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:19.127273+0000","last_peered":"2024-09-15T03:20:19.127273+0000","last_clean":"2024-09-15T03:20:19.127273+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:19.127273+0000","last_undegraded":"2024-09-15T03:20:19.127273+0000","last_fullsized":"2024-09-15T03:20:19.127273+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15965,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129140+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:19.129140+0000","last_peered":"2024-09-15T03:20:19.129140+0000","last_clean":"2024-09-15T03:20:19.129140+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:19.129140+0000","last_undegraded":"2024-09-15T03:20:19.129140+0000","last_fullsized":"2024-09-15T03:20:19.129140+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16332,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128740+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:19.128740+0000","last_peered":"2024-09-15T03:20:19.128740+0000","last_clean":"2024-09-15T03:20:19.128740+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:19.128740+0000","last_undegraded":"2024-09-15T03:20:19.128740+0000","last_fullsized":"2024-09-15T03:20:19.128740+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267276+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267276+0000","last_peered":"2024-09-15T03:20:20.267276+0000","last_clean":"2024-09-15T03:20:20.267276+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267276+0000","last_undegraded":"2024-09-15T03:20:20.267276+0000","last_fullsized":"2024-09-15T03:20:20.267276+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12013,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128929+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:19.128929+0000","last_peered":"2024-09-15T03:20:19.128929+0000","last_clean":"2024-09-15T03:20:19.128929+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:19.128929+0000","last_undegraded":"2024-09-15T03:20:19.128929+0000","last_fullsized":"2024-09-15T03:20:19.128929+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12147,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129049+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:19.129049+0000","last_peered":"2024-09-15T03:20:19.129049+0000","last_clean":"2024-09-15T03:20:19.129049+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:19.129049+0000","last_undegraded":"2024-09-15T03:20:19.129049+0000","last_fullsized":"2024-09-15T03:20:19.129049+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932228,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439612,"statfs":{"total":96636764160,"available":95682162688,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.497},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.67300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919292,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452548,"statfs":{"total":96636764160,"available":95695409152,"internally_reserved":0,"allocated":507904,"data_stored":221197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.439}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":930300,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":929659,"kb_avail":93441540,"statfs":{"total":96636764160,"available":95684136960,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":951971393},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":2.8900000000000001,"15min":2.3799999999999999},"min":{"1min":0.28499999999999998,"5min":0.17899999999999999,"15min":0.16900000000000001},"max":{"1min":0.61599999999999999,"5min":28.456,"15min":28.456},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.8560000000000001,"15min":1.9379999999999999},"min":{"1min":0.26500000000000001,"5min":0.19,"15min":0.187},"max":{"1min":0.77700000000000002,"5min":35.463000000000001,"15min":49.399999999999999},"last":0.58499999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":1.9670000000000001,"15min":2.0209999999999999},"min":{"1min":0.38900000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69699999999999995,"5min":25.661000000000001,"15min":26.911000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":2.3780000000000001,"15min":2.677},"min":{"1min":0.33300000000000002,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.70199999999999996,"5min":17.347999999999999,"15min":28.145},"last":0.70199999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:19:21 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":3.5870000000000002,"15min":2.5699999999999998},"min":{"1min":0.309,"5min":0.25,"15min":0.222},"max":{"1min":0.89300000000000002,"5min":34.875,"15min":59.225999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.626,"5min":2.1909999999999998,"15min":1.843},"min":{"1min":0.39800000000000002,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.93799999999999994,"5min":33.444000000000003,"15min":33.444000000000003},"last":0.54800000000000004}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919748,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452092,"statfs":{"total":96636764160,"available":95694942208,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:28.164 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:28.631 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:28.634 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:28.646 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-15T03:20:27.040441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705728,"kb_used_data":2124,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781632,"statfs":{"total":386547056640,"available":382752391168,"internally_reserved":0,"allocated":2174976,"data_stored":1018710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002289"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12939,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:22.405844+0000","last_change":"2024-09-15T03:20:19.046425+0000","last_active":"2024-09-15T03:20:22.405844+0000","last_peered":"2024-09-15T03:20:22.405844+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:22.405844+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14922,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129322+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:19.129322+0000","last_peered":"2024-09-15T03:20:19.129322+0000","last_clean":"2024-09-15T03:20:19.129322+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:19.129322+0000","last_undegraded":"2024-09-15T03:20:19.129322+0000","last_fullsized":"2024-09-15T03:20:19.129322+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127558+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:19.127558+0000","last_peered":"2024-09-15T03:20:19.127558+0000","last_clean":"2024-09-15T03:20:19.127558+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:19.127558+0000","last_undegraded":"2024-09-15T03:20:19.127558+0000","last_fullsized":"2024-09-15T03:20:19.127558+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15624,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128698+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:19.128698+0000","last_peered":"2024-09-15T03:20:19.128698+0000","last_clean":"2024-09-15T03:20:19.128698+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:19.128698+0000","last_undegraded":"2024-09-15T03:20:19.128698+0000","last_fullsized":"2024-09-15T03:20:19.128698+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16396,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125977+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:19.125977+0000","last_peered":"2024-09-15T03:20:19.125977+0000","last_clean":"2024-09-15T03:20:19.125977+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:19.125977+0000","last_undegraded":"2024-09-15T03:20:19.125977+0000","last_fullsized":"2024-09-15T03:20:19.125977+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16197,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127174+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:19.127174+0000","last_peered":"2024-09-15T03:20:19.127174+0000","last_clean":"2024-09-15T03:20:19.127174+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:19.127174+0000","last_undegraded":"2024-09-15T03:20:19.127174+0000","last_fullsized":"2024-09-15T03:20:19.127174+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16172,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.132113+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:19.132113+0000","last_peered":"2024-09-15T03:20:19.132113+0000","last_clean":"2024-09-15T03:20:19.132113+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:19.132113+0000","last_undegraded":"2024-09-15T03:20:19.132113+0000","last_fullsized":"2024-09-15T03:20:19.132113+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12235,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128900+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:19.128900+0000","last_peered":"2024-09-15T03:20:19.128900+0000","last_clean":"2024-09-15T03:20:19.128900+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:19.128900+0000","last_undegraded":"2024-09-15T03:20:19.128900+0000","last_fullsized":"2024-09-15T03:20:19.128900+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16156,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125999+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:19.125999+0000","last_peered":"2024-09-15T03:20:19.125999+0000","last_clean":"2024-09-15T03:20:19.125999+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:19.125999+0000","last_undegraded":"2024-09-15T03:20:19.125999+0000","last_fullsized":"2024-09-15T03:20:19.125999+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16212,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127708+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:19.127708+0000","last_peered":"2024-09-15T03:20:19.127708+0000","last_clean":"2024-09-15T03:20:19.127708+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:19.127708+0000","last_undegraded":"2024-09-15T03:20:19.127708+0000","last_fullsized":"2024-09-15T03:20:19.127708+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16143,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127273+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:19.127273+0000","last_peered":"2024-09-15T03:20:19.127273+0000","last_clean":"2024-09-15T03:20:19.127273+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:19.127273+0000","last_undegraded":"2024-09-15T03:20:19.127273+0000","last_fullsized":"2024-09-15T03:20:19.127273+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15965,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129140+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:19.129140+0000","last_peered":"2024-09-15T03:20:19.129140+0000","last_clean":"2024-09-15T03:20:19.129140+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:19.129140+0000","last_undegraded":"2024-09-15T03:20:19.129140+0000","last_fullsized":"2024-09-15T03:20:19.129140+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16332,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128740+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:19.128740+0000","last_peered":"2024-09-15T03:20:19.128740+0000","last_clean":"2024-09-15T03:20:19.128740+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:19.128740+0000","last_undegraded":"2024-09-15T03:20:19.128740+0000","last_fullsized":"2024-09-15T03:20:19.128740+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267276+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267276+0000","last_peered":"2024-09-15T03:20:20.267276+0000","last_clean":"2024-09-15T03:20:20.267276+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267276+0000","last_undegraded":"2024-09-15T03:20:20.267276+0000","last_fullsized":"2024-09-15T03:20:20.267276+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12013,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128929+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:19.128929+0000","last_peered":"2024-09-15T03:20:19.128929+0000","last_clean":"2024-09-15T03:20:19.128929+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:19.128929+0000","last_undegraded":"2024-09-15T03:20:19.128929+0000","last_fullsized":"2024-09-15T03:20:19.128929+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12147,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129049+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:19.129049+0000","last_peered":"2024-09-15T03:20:19.129049+0000","last_clean":"2024-09-15T03:20:19.129049+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:19.129049+0000","last_undegraded":"2024-09-15T03:20:19.129049+0000","last_fullsized":"2024-09-15T03:20:19.129049+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932228,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439612,"statfs":{"total":96636764160,"available":95682162688,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.497},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.67300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919292,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452548,"statfs":{"total":96636764160,"available":95695409152,"internally_reserved":0,"allocated":507904,"data_stored":221197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.439}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934460,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437380,"statfs":{"total":96636764160,"available":95679877120,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.53800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.35899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919748,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452092,"statfs":{"total":96636764160,"available":95694942208,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:28.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:28.950 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:28.954 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:28.968 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":528,"stamp":"2024-09-15T03:20:27.040441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705728,"kb_used_data":2124,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781632,"statfs":{"total":386547056640,"available":382752391168,"internally_reserved":0,"allocated":2174976,"data_stored":1018710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-769,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002289"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":12939,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:22.405844+0000","last_change":"2024-09-15T03:20:19.046425+0000","last_active":"2024-09-15T03:20:22.405844+0000","last_peered":"2024-09-15T03:20:22.405844+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:22.405844+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14922,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129322+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:19.129322+0000","last_peered":"2024-09-15T03:20:19.129322+0000","last_clean":"2024-09-15T03:20:19.129322+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:19.129322+0000","last_undegraded":"2024-09-15T03:20:19.129322+0000","last_fullsized":"2024-09-15T03:20:19.129322+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127558+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:19.127558+0000","last_peered":"2024-09-15T03:20:19.127558+0000","last_clean":"2024-09-15T03:20:19.127558+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:19.127558+0000","last_undegraded":"2024-09-15T03:20:19.127558+0000","last_fullsized":"2024-09-15T03:20:19.127558+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15624,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128698+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:19.128698+0000","last_peered":"2024-09-15T03:20:19.128698+0000","last_clean":"2024-09-15T03:20:19.128698+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:19.128698+0000","last_undegraded":"2024-09-15T03:20:19.128698+0000","last_fullsized":"2024-09-15T03:20:19.128698+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16396,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125977+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:19.125977+0000","last_peered":"2024-09-15T03:20:19.125977+0000","last_clean":"2024-09-15T03:20:19.125977+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:19.125977+0000","last_undegraded":"2024-09-15T03:20:19.125977+0000","last_fullsized":"2024-09-15T03:20:19.125977+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16197,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127174+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:19.127174+0000","last_peered":"2024-09-15T03:20:19.127174+0000","last_clean":"2024-09-15T03:20:19.127174+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:19.127174+0000","last_undegraded":"2024-09-15T03:20:19.127174+0000","last_fullsized":"2024-09-15T03:20:19.127174+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16172,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.132113+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:19.132113+0000","last_peered":"2024-09-15T03:20:19.132113+0000","last_clean":"2024-09-15T03:20:19.132113+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:19.132113+0000","last_undegraded":"2024-09-15T03:20:19.132113+0000","last_fullsized":"2024-09-15T03:20:19.132113+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12235,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128900+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:19.128900+0000","last_peered":"2024-09-15T03:20:19.128900+0000","last_clean":"2024-09-15T03:20:19.128900+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:19.128900+0000","last_undegraded":"2024-09-15T03:20:19.128900+0000","last_fullsized":"2024-09-15T03:20:19.128900+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16156,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125999+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:19.125999+0000","last_peered":"2024-09-15T03:20:19.125999+0000","last_clean":"2024-09-15T03:20:19.125999+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:19.125999+0000","last_undegraded":"2024-09-15T03:20:19.125999+0000","last_fullsized":"2024-09-15T03:20:19.125999+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16212,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127708+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:19.127708+0000","last_peered":"2024-09-15T03:20:19.127708+0000","last_clean":"2024-09-15T03:20:19.127708+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:19.127708+0000","last_undegraded":"2024-09-15T03:20:19.127708+0000","last_fullsized":"2024-09-15T03:20:19.127708+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16143,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127273+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:19.127273+0000","last_peered":"2024-09-15T03:20:19.127273+0000","last_clean":"2024-09-15T03:20:19.127273+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:19.127273+0000","last_undegraded":"2024-09-15T03:20:19.127273+0000","last_fullsized":"2024-09-15T03:20:19.127273+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15965,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129140+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:19.129140+0000","last_peered":"2024-09-15T03:20:19.129140+0000","last_clean":"2024-09-15T03:20:19.129140+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:19.129140+0000","last_undegraded":"2024-09-15T03:20:19.129140+0000","last_fullsized":"2024-09-15T03:20:19.129140+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16332,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128740+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:19.128740+0000","last_peered":"2024-09-15T03:20:19.128740+0000","last_clean":"2024-09-15T03:20:19.128740+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:19.128740+0000","last_undegraded":"2024-09-15T03:20:19.128740+0000","last_fullsized":"2024-09-15T03:20:19.128740+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267276+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267276+0000","last_peered":"2024-09-15T03:20:20.267276+0000","last_clean":"2024-09-15T03:20:20.267276+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267276+0000","last_undegraded":"2024-09-15T03:20:20.267276+0000","last_fullsized":"2024-09-15T03:20:20.267276+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12013,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128929+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:19.128929+0000","last_peered":"2024-09-15T03:20:19.128929+0000","last_clean":"2024-09-15T03:20:19.128929+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:19.128929+0000","last_undegraded":"2024-09-15T03:20:19.128929+0000","last_fullsized":"2024-09-15T03:20:19.128929+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12147,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129049+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:19.129049+0000","last_peered":"2024-09-15T03:20:19.129049+0000","last_clean":"2024-09-15T03:20:19.129049+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:19.129049+0000","last_undegraded":"2024-09-15T03:20:19.129049+0000","last_fullsized":"2024-09-15T03:20:19.129049+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27192,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932228,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439612,"statfs":{"total":96636764160,"available":95682162688,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.497},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.67300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919292,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452548,"statfs":{"total":96636764160,"available":95695409152,"internally_reserved":0,"allocated":507904,"data_stored":221197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.41199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.439}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.51300000000000001}]}]},{"osd":1,"up_from":29,"seq":124554051755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934460,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437380,"statfs":{"total":96636764160,"available":95679877120,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.53800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.35899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247994,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919748,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452092,"statfs":{"total":96636764160,"available":95694942208,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T03:20:28.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11910", 'reported_seq': 12939, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T03:20:22.405844+0000', 'last_change': '2024-09-15T03:20:19.046425+0000', 'last_active': '2024-09-15T03:20:22.405844+0000', 'last_peered': '2024-09-15T03:20:22.405844+0000', 'last_clean': '2024-09-15T03:04:53.732736+0000', 'last_became_active': '2024-09-15T03:06:09.284160+0000', 'last_became_peered': '2024-09-15T03:06:09.284160+0000', 'last_unstale': '2024-09-15T03:20:22.405844+0000', 'last_undegraded': '2024-09-15T03:06:08.918769+0000', 'last_fullsized': '2024-09-15T03:06:08.835658+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'last_clean_scrub_stamp': '2024-09-15T03:04:53.732736+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1498, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 478, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1498}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T03:20:28.969 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:20:29.352 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:29.352 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 637/112 objects degraded (568.750%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":952,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":195132,"bytes_used":3794665472,"bytes_avail":382752391168,"bytes_total":386547056640,"degraded_objects":637,"degraded_total":112,"degraded_ratio":5.6875},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:20:29.366 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T03:20:29.366 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:29.663 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:29.666 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:29.675 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":529,"stamp":"2024-09-15T03:20:29.040993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28053,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3705728,"kb_used_data":2124,"kb_used_omap":14,"kb_used_meta":3703537,"kb_avail":373781632,"statfs":{"total":386547056640,"available":382752391168,"internally_reserved":0,"allocated":2174976,"data_stored":1018710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3792421915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.916515"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":14661,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:27.409046+0000","last_change":"2024-09-15T03:20:19.046425+0000","last_active":"2024-09-15T03:20:27.409046+0000","last_peered":"2024-09-15T03:20:27.409046+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:27.409046+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1339,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14922,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129322+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:19.129322+0000","last_peered":"2024-09-15T03:20:19.129322+0000","last_clean":"2024-09-15T03:20:19.129322+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:19.129322+0000","last_undegraded":"2024-09-15T03:20:19.129322+0000","last_fullsized":"2024-09-15T03:20:19.129322+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127558+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:19.127558+0000","last_peered":"2024-09-15T03:20:19.127558+0000","last_clean":"2024-09-15T03:20:19.127558+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:19.127558+0000","last_undegraded":"2024-09-15T03:20:19.127558+0000","last_fullsized":"2024-09-15T03:20:19.127558+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15624,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128698+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:19.128698+0000","last_peered":"2024-09-15T03:20:19.128698+0000","last_clean":"2024-09-15T03:20:19.128698+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:19.128698+0000","last_undegraded":"2024-09-15T03:20:19.128698+0000","last_fullsized":"2024-09-15T03:20:19.128698+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16396,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125977+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:19.125977+0000","last_peered":"2024-09-15T03:20:19.125977+0000","last_clean":"2024-09-15T03:20:19.125977+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:19.125977+0000","last_undegraded":"2024-09-15T03:20:19.125977+0000","last_fullsized":"2024-09-15T03:20:19.125977+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16197,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127174+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:19.127174+0000","last_peered":"2024-09-15T03:20:19.127174+0000","last_clean":"2024-09-15T03:20:19.127174+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:19.127174+0000","last_undegraded":"2024-09-15T03:20:19.127174+0000","last_fullsized":"2024-09-15T03:20:19.127174+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16172,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.132113+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:19.132113+0000","last_peered":"2024-09-15T03:20:19.132113+0000","last_clean":"2024-09-15T03:20:19.132113+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:19.132113+0000","last_undegraded":"2024-09-15T03:20:19.132113+0000","last_fullsized":"2024-09-15T03:20:19.132113+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12235,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128900+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:19.128900+0000","last_peered":"2024-09-15T03:20:19.128900+0000","last_clean":"2024-09-15T03:20:19.128900+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:19.128900+0000","last_undegraded":"2024-09-15T03:20:19.128900+0000","last_fullsized":"2024-09-15T03:20:19.128900+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16156,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125999+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:19.125999+0000","last_peered":"2024-09-15T03:20:19.125999+0000","last_clean":"2024-09-15T03:20:19.125999+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:19.125999+0000","last_undegraded":"2024-09-15T03:20:19.125999+0000","last_fullsized":"2024-09-15T03:20:19.125999+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16212,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127708+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:19.127708+0000","last_peered":"2024-09-15T03:20:19.127708+0000","last_clean":"2024-09-15T03:20:19.127708+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:19.127708+0000","last_undegraded":"2024-09-15T03:20:19.127708+0000","last_fullsized":"2024-09-15T03:20:19.127708+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16143,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127273+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:19.127273+0000","last_peered":"2024-09-15T03:20:19.127273+0000","last_clean":"2024-09-15T03:20:19.127273+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:19.127273+0000","last_undegraded":"2024-09-15T03:20:19.127273+0000","last_fullsized":"2024-09-15T03:20:19.127273+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15965,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129140+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:19.129140+0000","last_peered":"2024-09-15T03:20:19.129140+0000","last_clean":"2024-09-15T03:20:19.129140+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:19.129140+0000","last_undegraded":"2024-09-15T03:20:19.129140+0000","last_fullsized":"2024-09-15T03:20:19.129140+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16332,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128740+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:19.128740+0000","last_peered":"2024-09-15T03:20:19.128740+0000","last_clean":"2024-09-15T03:20:19.128740+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:19.128740+0000","last_undegraded":"2024-09-15T03:20:19.128740+0000","last_fullsized":"2024-09-15T03:20:19.128740+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11797,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267276+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267276+0000","last_peered":"2024-09-15T03:20:20.267276+0000","last_clean":"2024-09-15T03:20:20.267276+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267276+0000","last_undegraded":"2024-09-15T03:20:20.267276+0000","last_fullsized":"2024-09-15T03:20:20.267276+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12013,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.128929+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:19.128929+0000","last_peered":"2024-09-15T03:20:19.128929+0000","last_clean":"2024-09-15T03:20:19.128929+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:19.128929+0000","last_undegraded":"2024-09-15T03:20:19.128929+0000","last_fullsized":"2024-09-15T03:20:19.128929+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12147,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.129049+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:19.129049+0000","last_peered":"2024-09-15T03:20:19.129049+0000","last_clean":"2024-09-15T03:20:19.129049+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:19.129049+0000","last_undegraded":"2024-09-15T03:20:19.129049+0000","last_fullsized":"2024-09-15T03:20:19.129049+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28053,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932228,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439612,"statfs":{"total":96636764160,"available":95682162688,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.36199999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.66900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919292,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452548,"statfs":{"total":96636764160,"available":95695409152,"internally_reserved":0,"allocated":507904,"data_stored":221197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.627}]}]},{"osd":1,"up_from":29,"seq":124554051755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":934460,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":933819,"kb_avail":93437380,"statfs":{"total":96636764160,"available":95679877120,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":956231233},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.28599999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.53800000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.35899999999999999}]}]},{"osd":0,"up_from":23,"seq":98784247995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919748,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452092,"statfs":{"total":96636764160,"available":95694942208,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:32.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:32.981 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:32.992 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2024-09-15T03:20:32.175246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28053,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3697408,"kb_used_data":2124,"kb_used_omap":14,"kb_used_meta":3695217,"kb_avail":373789952,"statfs":{"total":386547056640,"available":382760910848,"internally_reserved":0,"allocated":2174976,"data_stored":1018710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3783902235},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-861,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":861,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.135820"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":14661,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T03:20:27.409046+0000","last_change":"2024-09-15T03:20:19.046425+0000","last_active":"2024-09-15T03:20:27.409046+0000","last_peered":"2024-09-15T03:20:27.409046+0000","last_clean":"2024-09-15T03:04:53.732736+0000","last_became_active":"2024-09-15T03:06:09.284160+0000","last_became_peered":"2024-09-15T03:06:09.284160+0000","last_unstale":"2024-09-15T03:20:27.409046+0000","last_undegraded":"2024-09-15T03:06:08.918769+0000","last_fullsized":"2024-09-15T03:06:08.835658+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:49:40.949175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1339,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-637}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14923,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127609+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:20.127609+0000","last_peered":"2024-09-15T03:20:20.127609+0000","last_clean":"2024-09-15T03:20:20.127609+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:20.127609+0000","last_undegraded":"2024-09-15T03:20:20.127609+0000","last_fullsized":"2024-09-15T03:20:20.127609+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16551,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127558+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:19.127558+0000","last_peered":"2024-09-15T03:20:19.127558+0000","last_clean":"2024-09-15T03:20:19.127558+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:19.127558+0000","last_undegraded":"2024-09-15T03:20:19.127558+0000","last_fullsized":"2024-09-15T03:20:19.127558+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15625,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127157+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:20.127157+0000","last_peered":"2024-09-15T03:20:20.127157+0000","last_clean":"2024-09-15T03:20:20.127157+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:20.127157+0000","last_undegraded":"2024-09-15T03:20:20.127157+0000","last_fullsized":"2024-09-15T03:20:20.127157+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16396,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125977+0000","last_change":"2024-09-15T03:19:02.961706+0000","last_active":"2024-09-15T03:20:19.125977+0000","last_peered":"2024-09-15T03:20:19.125977+0000","last_clean":"2024-09-15T03:20:19.125977+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:19.125977+0000","last_undegraded":"2024-09-15T03:20:19.125977+0000","last_fullsized":"2024-09-15T03:20:19.125977+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":48317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16197,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127174+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:19.127174+0000","last_peered":"2024-09-15T03:20:19.127174+0000","last_clean":"2024-09-15T03:20:19.127174+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:19.127174+0000","last_undegraded":"2024-09-15T03:20:19.127174+0000","last_fullsized":"2024-09-15T03:20:19.127174+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16172,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.132113+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:19.132113+0000","last_peered":"2024-09-15T03:20:19.132113+0000","last_clean":"2024-09-15T03:20:19.132113+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:19.132113+0000","last_undegraded":"2024-09-15T03:20:19.132113+0000","last_fullsized":"2024-09-15T03:20:19.132113+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12236,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127375+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:20.127375+0000","last_peered":"2024-09-15T03:20:20.127375+0000","last_clean":"2024-09-15T03:20:20.127375+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:20.127375+0000","last_undegraded":"2024-09-15T03:20:20.127375+0000","last_fullsized":"2024-09-15T03:20:20.127375+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16156,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.125999+0000","last_change":"2024-09-15T03:18:07.256839+0000","last_active":"2024-09-15T03:20:19.125999+0000","last_peered":"2024-09-15T03:20:19.125999+0000","last_clean":"2024-09-15T03:20:19.125999+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:19.125999+0000","last_undegraded":"2024-09-15T03:20:19.125999+0000","last_fullsized":"2024-09-15T03:20:19.125999+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27876,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":47865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2048,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16212,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127708+0000","last_change":"2024-09-15T03:18:37.340382+0000","last_active":"2024-09-15T03:20:19.127708+0000","last_peered":"2024-09-15T03:20:19.127708+0000","last_clean":"2024-09-15T03:20:19.127708+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:19.127708+0000","last_undegraded":"2024-09-15T03:20:19.127708+0000","last_fullsized":"2024-09-15T03:20:19.127708+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":47862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16143,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T03:20:19.127273+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:19.127273+0000","last_peered":"2024-09-15T03:20:19.127273+0000","last_clean":"2024-09-15T03:20:19.127273+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:19.127273+0000","last_undegraded":"2024-09-15T03:20:19.127273+0000","last_fullsized":"2024-09-15T03:20:19.127273+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15966,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127528+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:20.127528+0000","last_peered":"2024-09-15T03:20:20.127528+0000","last_clean":"2024-09-15T03:20:20.127528+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:20.127528+0000","last_undegraded":"2024-09-15T03:20:20.127528+0000","last_fullsized":"2024-09-15T03:20:20.127528+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16333,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127225+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:20.127225+0000","last_peered":"2024-09-15T03:20:20.127225+0000","last_clean":"2024-09-15T03:20:20.127225+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:20.127225+0000","last_undegraded":"2024-09-15T03:20:20.127225+0000","last_fullsized":"2024-09-15T03:20:20.127225+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267844+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267844+0000","last_peered":"2024-09-15T03:20:20.267844+0000","last_clean":"2024-09-15T03:20:20.267844+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267844+0000","last_undegraded":"2024-09-15T03:20:20.267844+0000","last_fullsized":"2024-09-15T03:20:20.267844+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12014,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127592+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:20.127592+0000","last_peered":"2024-09-15T03:20:20.127592+0000","last_clean":"2024-09-15T03:20:20.127592+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:20.127592+0000","last_undegraded":"2024-09-15T03:20:20.127592+0000","last_fullsized":"2024-09-15T03:20:20.127592+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12148,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127677+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:20.127677+0000","last_peered":"2024-09-15T03:20:20.127677+0000","last_clean":"2024-09-15T03:20:20.127677+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:20.127677+0000","last_undegraded":"2024-09-15T03:20:20.127677+0000","last_fullsized":"2024-09-15T03:20:20.127677+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195132,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190679,"num_write_kb":762811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28053,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932228,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439612,"statfs":{"total":96636764160,"available":95682162688,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.36199999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.66900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919292,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452548,"statfs":{"total":96636764160,"available":95695409152,"internally_reserved":0,"allocated":507904,"data_stored":221197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":2.0630000000000002,"15min":1.7470000000000001},"min":{"1min":0.35399999999999998,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.89000000000000001,"5min":15.833,"15min":32.295999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":2.1379999999999999,"15min":1.8160000000000001},"min":{"1min":0.44600000000000001,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.94599999999999995,"5min":15.872999999999999,"15min":32.340000000000003},"last":0.56599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.627}]}]},{"osd":1,"up_from":29,"seq":124554051756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926140,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445700,"statfs":{"total":96636764160,"available":95688396800,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.67700000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.57199999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247995,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919748,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":919227,"kb_avail":93452092,"statfs":{"total":96636764160,"available":95694942208,"internally_reserved":0,"allocated":512000,"data_stored":222147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":941289025},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.52800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.627}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:32.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:33.298 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:33.300 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:33.312 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-09-15T03:20:33.042293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146349,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154675,"num_write_kb":618767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22469,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701596,"kb_used_data":2156,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785764,"statfs":{"total":386547056640,"available":382756622336,"internally_reserved":0,"allocated":2207744,"data_stored":1032865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-48783,"num_objects":-7,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-36004,"num_write_kb":-144044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002292"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14923,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127609+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:20.127609+0000","last_peered":"2024-09-15T03:20:20.127609+0000","last_clean":"2024-09-15T03:20:20.127609+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:20.127609+0000","last_undegraded":"2024-09-15T03:20:20.127609+0000","last_fullsized":"2024-09-15T03:20:20.127609+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173367+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:31.173367+0000","last_peered":"2024-09-15T03:20:31.173367+0000","last_clean":"2024-09-15T03:20:31.173367+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:31.173367+0000","last_undegraded":"2024-09-15T03:20:31.173367+0000","last_fullsized":"2024-09-15T03:20:31.173367+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15625,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127157+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:20.127157+0000","last_peered":"2024-09-15T03:20:20.127157+0000","last_clean":"2024-09-15T03:20:20.127157+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:20.127157+0000","last_undegraded":"2024-09-15T03:20:20.127157+0000","last_fullsized":"2024-09-15T03:20:20.127157+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16174,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.183141+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:31.183141+0000","last_peered":"2024-09-15T03:20:31.183141+0000","last_clean":"2024-09-15T03:20:31.183141+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:31.183141+0000","last_undegraded":"2024-09-15T03:20:31.183141+0000","last_fullsized":"2024-09-15T03:20:31.183141+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12236,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127375+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:20.127375+0000","last_peered":"2024-09-15T03:20:20.127375+0000","last_clean":"2024-09-15T03:20:20.127375+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:20.127375+0000","last_undegraded":"2024-09-15T03:20:20.127375+0000","last_fullsized":"2024-09-15T03:20:20.127375+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16145,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173802+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:31.173802+0000","last_peered":"2024-09-15T03:20:31.173802+0000","last_clean":"2024-09-15T03:20:31.173802+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:31.173802+0000","last_undegraded":"2024-09-15T03:20:31.173802+0000","last_fullsized":"2024-09-15T03:20:31.173802+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15966,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127528+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:20.127528+0000","last_peered":"2024-09-15T03:20:20.127528+0000","last_clean":"2024-09-15T03:20:20.127528+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:20.127528+0000","last_undegraded":"2024-09-15T03:20:20.127528+0000","last_fullsized":"2024-09-15T03:20:20.127528+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16333,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127225+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:20.127225+0000","last_peered":"2024-09-15T03:20:20.127225+0000","last_clean":"2024-09-15T03:20:20.127225+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:20.127225+0000","last_undegraded":"2024-09-15T03:20:20.127225+0000","last_fullsized":"2024-09-15T03:20:20.127225+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267844+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267844+0000","last_peered":"2024-09-15T03:20:20.267844+0000","last_clean":"2024-09-15T03:20:20.267844+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267844+0000","last_undegraded":"2024-09-15T03:20:20.267844+0000","last_fullsized":"2024-09-15T03:20:20.267844+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12014,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127592+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:20.127592+0000","last_peered":"2024-09-15T03:20:20.127592+0000","last_clean":"2024-09-15T03:20:20.127592+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:20.127592+0000","last_undegraded":"2024-09-15T03:20:20.127592+0000","last_fullsized":"2024-09-15T03:20:20.127592+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12148,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127677+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:20.127677+0000","last_peered":"2024-09-15T03:20:20.127677+0000","last_clean":"2024-09-15T03:20:20.127677+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:20.127677+0000","last_undegraded":"2024-09-15T03:20:20.127677+0000","last_fullsized":"2024-09-15T03:20:20.127677+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":146349,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154675,"num_write_kb":618767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22469,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932244,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439596,"statfs":{"total":96636764160,"available":95682146304,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919300,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452540,"statfs":{"total":96636764160,"available":95695400960,"internally_reserved":0,"allocated":516096,"data_stored":224741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.65900000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926140,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445700,"statfs":{"total":96636764160,"available":95688396800,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.67700000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.57199999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923912,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447928,"statfs":{"total":96636764160,"available":95690678272,"internally_reserved":0,"allocated":520192,"data_stored":225691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.501},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.746}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:33.313 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T03:20:33.663 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:33.664 INFO:teuthology.orchestra.run.smithi060.stdout:{"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":956,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1726369566,"num_in_osds":4,"osd_in_since":1726369479,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16}],"num_pgs":16,"num_pools":1,"num_objects":21,"data_bytes":146349,"bytes_used":3790434304,"bytes_avail":382756622336,"bytes_total":386547056640},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-15T03:13:44.876539+0000","services":{}},"progress_events":{}} 2024-09-15T03:20:33.678 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T03:20:33.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:33.978 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:33.980 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:33.991 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-09-15T03:20:33.042293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146349,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154675,"num_write_kb":618767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22469,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701596,"kb_used_data":2156,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785764,"statfs":{"total":386547056640,"available":382756622336,"internally_reserved":0,"allocated":2207744,"data_stored":1032865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-48783,"num_objects":-7,"num_object_clones":0,"num_object_copies":-28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1498,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-36004,"num_write_kb":-144044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4723,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002292"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14923,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127609+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:20.127609+0000","last_peered":"2024-09-15T03:20:20.127609+0000","last_clean":"2024-09-15T03:20:20.127609+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:20.127609+0000","last_undegraded":"2024-09-15T03:20:20.127609+0000","last_fullsized":"2024-09-15T03:20:20.127609+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173367+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:31.173367+0000","last_peered":"2024-09-15T03:20:31.173367+0000","last_clean":"2024-09-15T03:20:31.173367+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:31.173367+0000","last_undegraded":"2024-09-15T03:20:31.173367+0000","last_fullsized":"2024-09-15T03:20:31.173367+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15625,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127157+0000","last_change":"2024-09-15T03:12:00.296816+0000","last_active":"2024-09-15T03:20:20.127157+0000","last_peered":"2024-09-15T03:20:20.127157+0000","last_clean":"2024-09-15T03:20:20.127157+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:20.127157+0000","last_undegraded":"2024-09-15T03:20:20.127157+0000","last_fullsized":"2024-09-15T03:20:20.127157+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11624,"num_write_kb":46501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1947,"num_bytes_recovered":14589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16174,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.183141+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:31.183141+0000","last_peered":"2024-09-15T03:20:31.183141+0000","last_clean":"2024-09-15T03:20:31.183141+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:31.183141+0000","last_undegraded":"2024-09-15T03:20:31.183141+0000","last_fullsized":"2024-09-15T03:20:31.183141+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12236,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127375+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:20.127375+0000","last_peered":"2024-09-15T03:20:20.127375+0000","last_clean":"2024-09-15T03:20:20.127375+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:20.127375+0000","last_undegraded":"2024-09-15T03:20:20.127375+0000","last_fullsized":"2024-09-15T03:20:20.127375+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16145,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173802+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:31.173802+0000","last_peered":"2024-09-15T03:20:31.173802+0000","last_clean":"2024-09-15T03:20:31.173802+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:31.173802+0000","last_undegraded":"2024-09-15T03:20:31.173802+0000","last_fullsized":"2024-09-15T03:20:31.173802+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15966,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127528+0000","last_change":"2024-09-15T03:09:41.991879+0000","last_active":"2024-09-15T03:20:20.127528+0000","last_peered":"2024-09-15T03:20:20.127528+0000","last_clean":"2024-09-15T03:20:20.127528+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:20.127528+0000","last_undegraded":"2024-09-15T03:20:20.127528+0000","last_fullsized":"2024-09-15T03:20:20.127528+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":47591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1994,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16333,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127225+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:20.127225+0000","last_peered":"2024-09-15T03:20:20.127225+0000","last_clean":"2024-09-15T03:20:20.127225+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:20.127225+0000","last_undegraded":"2024-09-15T03:20:20.127225+0000","last_fullsized":"2024-09-15T03:20:20.127225+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11798,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.267844+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:20.267844+0000","last_peered":"2024-09-15T03:20:20.267844+0000","last_clean":"2024-09-15T03:20:20.267844+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:20.267844+0000","last_undegraded":"2024-09-15T03:20:20.267844+0000","last_fullsized":"2024-09-15T03:20:20.267844+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12014,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127592+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:20.127592+0000","last_peered":"2024-09-15T03:20:20.127592+0000","last_clean":"2024-09-15T03:20:20.127592+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:20.127592+0000","last_undegraded":"2024-09-15T03:20:20.127592+0000","last_fullsized":"2024-09-15T03:20:20.127592+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12148,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T03:20:20.127677+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:20.127677+0000","last_peered":"2024-09-15T03:20:20.127677+0000","last_clean":"2024-09-15T03:20:20.127677+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:20.127677+0000","last_undegraded":"2024-09-15T03:20:20.127677+0000","last_fullsized":"2024-09-15T03:20:20.127677+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":146349,"num_objects":21,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154675,"num_write_kb":618767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22469,"num_bytes_recovered":31072256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932244,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439596,"statfs":{"total":96636764160,"available":95682146304,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919300,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452540,"statfs":{"total":96636764160,"available":95695400960,"internally_reserved":0,"allocated":516096,"data_stored":224741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.65900000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926140,"kb_used_data":628,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445700,"statfs":{"total":96636764160,"available":95688396800,"internally_reserved":0,"allocated":643072,"data_stored":353219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.67700000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.57199999999999995}]}]},{"osd":0,"up_from":23,"seq":98784247996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923912,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447928,"statfs":{"total":96636764160,"available":95690678272,"internally_reserved":0,"allocated":520192,"data_stored":225691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.501},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.746}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:36.993 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:37.301 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:37.315 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-09-15T03:20:37.043359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701612,"kb_used_data":2172,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785748,"statfs":{"total":386547056640,"available":382756605952,"internally_reserved":0,"allocated":2224128,"data_stored":1039932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-69690,"num_objects":-10,"num_object_clones":0,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-59525,"num_write_kb":-238136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9525,"num_bytes_recovered":-30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002365"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14924,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175859+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:31.175859+0000","last_peered":"2024-09-15T03:20:31.175859+0000","last_clean":"2024-09-15T03:20:31.175859+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:31.175859+0000","last_undegraded":"2024-09-15T03:20:31.175859+0000","last_fullsized":"2024-09-15T03:20:31.175859+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173367+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:31.173367+0000","last_peered":"2024-09-15T03:20:31.173367+0000","last_clean":"2024-09-15T03:20:31.173367+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:31.173367+0000","last_undegraded":"2024-09-15T03:20:31.173367+0000","last_fullsized":"2024-09-15T03:20:31.173367+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15636,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:35.171317+0000","last_change":"2024-09-15T03:20:35.171084+0000","last_active":"2024-09-15T03:20:35.171317+0000","last_peered":"2024-09-15T03:20:35.171317+0000","last_clean":"2024-09-15T03:20:35.171317+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:35.171317+0000","last_undegraded":"2024-09-15T03:20:35.171317+0000","last_fullsized":"2024-09-15T03:20:35.171317+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:35.170789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:35.170789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:25.514810+0000","scrub_duration":0.027974141000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16174,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.183141+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:31.183141+0000","last_peered":"2024-09-15T03:20:31.183141+0000","last_clean":"2024-09-15T03:20:31.183141+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:31.183141+0000","last_undegraded":"2024-09-15T03:20:31.183141+0000","last_fullsized":"2024-09-15T03:20:31.183141+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175659+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:31.175659+0000","last_peered":"2024-09-15T03:20:31.175659+0000","last_clean":"2024-09-15T03:20:31.175659+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:31.175659+0000","last_undegraded":"2024-09-15T03:20:31.175659+0000","last_fullsized":"2024-09-15T03:20:31.175659+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16145,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173802+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:31.173802+0000","last_peered":"2024-09-15T03:20:31.173802+0000","last_clean":"2024-09-15T03:20:31.173802+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:31.173802+0000","last_undegraded":"2024-09-15T03:20:31.173802+0000","last_fullsized":"2024-09-15T03:20:31.173802+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15977,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.188829+0000","last_change":"2024-09-15T03:20:32.188544+0000","last_active":"2024-09-15T03:20:32.188829+0000","last_peered":"2024-09-15T03:20:32.188829+0000","last_clean":"2024-09-15T03:20:32.188829+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:32.188829+0000","last_undegraded":"2024-09-15T03:20:32.188829+0000","last_fullsized":"2024-09-15T03:20:32.188829+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:32.188277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.188277+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:53.497786+0000","scrub_duration":0.031810063999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16334,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175447+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:31.175447+0000","last_peered":"2024-09-15T03:20:31.175447+0000","last_clean":"2024-09-15T03:20:31.175447+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:31.175447+0000","last_undegraded":"2024-09-15T03:20:31.175447+0000","last_fullsized":"2024-09-15T03:20:31.175447+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175573+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:31.175573+0000","last_peered":"2024-09-15T03:20:31.175573+0000","last_clean":"2024-09-15T03:20:31.175573+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:31.175573+0000","last_undegraded":"2024-09-15T03:20:31.175573+0000","last_fullsized":"2024-09-15T03:20:31.175573+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12015,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175615+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:31.175615+0000","last_peered":"2024-09-15T03:20:31.175615+0000","last_clean":"2024-09-15T03:20:31.175615+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:31.175615+0000","last_undegraded":"2024-09-15T03:20:31.175615+0000","last_fullsized":"2024-09-15T03:20:31.175615+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175777+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:31.175777+0000","last_peered":"2024-09-15T03:20:31.175777+0000","last_clean":"2024-09-15T03:20:31.175777+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:31.175777+0000","last_undegraded":"2024-09-15T03:20:31.175777+0000","last_fullsized":"2024-09-15T03:20:31.175777+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932244,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439596,"statfs":{"total":96636764160,"available":95682146304,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919300,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452540,"statfs":{"total":96636764160,"available":95695400960,"internally_reserved":0,"allocated":516096,"data_stored":224741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.65900000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.57599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.64200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923912,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447928,"statfs":{"total":96636764160,"available":95690678272,"internally_reserved":0,"allocated":520192,"data_stored":225691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.501},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.746}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:37.316 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-15T03:20:37.316 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-15T03:20:37.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:37.604 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:37.607 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:37.623 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-09-15T03:20:37.043359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701612,"kb_used_data":2172,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785748,"statfs":{"total":386547056640,"available":382756605952,"internally_reserved":0,"allocated":2224128,"data_stored":1039932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-69690,"num_objects":-10,"num_object_clones":0,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-59525,"num_write_kb":-238136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9525,"num_bytes_recovered":-30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002365"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14924,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175859+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:31.175859+0000","last_peered":"2024-09-15T03:20:31.175859+0000","last_clean":"2024-09-15T03:20:31.175859+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:31.175859+0000","last_undegraded":"2024-09-15T03:20:31.175859+0000","last_fullsized":"2024-09-15T03:20:31.175859+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173367+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:31.173367+0000","last_peered":"2024-09-15T03:20:31.173367+0000","last_clean":"2024-09-15T03:20:31.173367+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:31.173367+0000","last_undegraded":"2024-09-15T03:20:31.173367+0000","last_fullsized":"2024-09-15T03:20:31.173367+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15636,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:35.171317+0000","last_change":"2024-09-15T03:20:35.171084+0000","last_active":"2024-09-15T03:20:35.171317+0000","last_peered":"2024-09-15T03:20:35.171317+0000","last_clean":"2024-09-15T03:20:35.171317+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:35.171317+0000","last_undegraded":"2024-09-15T03:20:35.171317+0000","last_fullsized":"2024-09-15T03:20:35.171317+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:35.170789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:35.170789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:25.514810+0000","scrub_duration":0.027974141000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16174,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.183141+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:31.183141+0000","last_peered":"2024-09-15T03:20:31.183141+0000","last_clean":"2024-09-15T03:20:31.183141+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:31.183141+0000","last_undegraded":"2024-09-15T03:20:31.183141+0000","last_fullsized":"2024-09-15T03:20:31.183141+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175659+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:31.175659+0000","last_peered":"2024-09-15T03:20:31.175659+0000","last_clean":"2024-09-15T03:20:31.175659+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:31.175659+0000","last_undegraded":"2024-09-15T03:20:31.175659+0000","last_fullsized":"2024-09-15T03:20:31.175659+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16145,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173802+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:31.173802+0000","last_peered":"2024-09-15T03:20:31.173802+0000","last_clean":"2024-09-15T03:20:31.173802+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:31.173802+0000","last_undegraded":"2024-09-15T03:20:31.173802+0000","last_fullsized":"2024-09-15T03:20:31.173802+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15977,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.188829+0000","last_change":"2024-09-15T03:20:32.188544+0000","last_active":"2024-09-15T03:20:32.188829+0000","last_peered":"2024-09-15T03:20:32.188829+0000","last_clean":"2024-09-15T03:20:32.188829+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:32.188829+0000","last_undegraded":"2024-09-15T03:20:32.188829+0000","last_fullsized":"2024-09-15T03:20:32.188829+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:32.188277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.188277+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:53.497786+0000","scrub_duration":0.031810063999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16334,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175447+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:31.175447+0000","last_peered":"2024-09-15T03:20:31.175447+0000","last_clean":"2024-09-15T03:20:31.175447+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:31.175447+0000","last_undegraded":"2024-09-15T03:20:31.175447+0000","last_fullsized":"2024-09-15T03:20:31.175447+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175573+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:31.175573+0000","last_peered":"2024-09-15T03:20:31.175573+0000","last_clean":"2024-09-15T03:20:31.175573+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:31.175573+0000","last_undegraded":"2024-09-15T03:20:31.175573+0000","last_fullsized":"2024-09-15T03:20:31.175573+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12015,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175615+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:31.175615+0000","last_peered":"2024-09-15T03:20:31.175615+0000","last_clean":"2024-09-15T03:20:31.175615+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:31.175615+0000","last_undegraded":"2024-09-15T03:20:31.175615+0000","last_fullsized":"2024-09-15T03:20:31.175615+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175777+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:31.175777+0000","last_peered":"2024-09-15T03:20:31.175777+0000","last_clean":"2024-09-15T03:20:31.175777+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:31.175777+0000","last_undegraded":"2024-09-15T03:20:31.175777+0000","last_fullsized":"2024-09-15T03:20:31.175777+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932244,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439596,"statfs":{"total":96636764160,"available":95682146304,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919300,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452540,"statfs":{"total":96636764160,"available":95695400960,"internally_reserved":0,"allocated":516096,"data_stored":224741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.65900000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.57599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.64200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923912,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447928,"statfs":{"total":96636764160,"available":95690678272,"internally_reserved":0,"allocated":520192,"data_stored":225691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.501},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.746}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:37.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:37.906 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:37.910 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:37.920 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-09-15T03:20:37.043359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701612,"kb_used_data":2172,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785748,"statfs":{"total":386547056640,"available":382756605952,"internally_reserved":0,"allocated":2224128,"data_stored":1039932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-69690,"num_objects":-10,"num_object_clones":0,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-59525,"num_write_kb":-238136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9525,"num_bytes_recovered":-30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002365"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14924,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175859+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:31.175859+0000","last_peered":"2024-09-15T03:20:31.175859+0000","last_clean":"2024-09-15T03:20:31.175859+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:31.175859+0000","last_undegraded":"2024-09-15T03:20:31.175859+0000","last_fullsized":"2024-09-15T03:20:31.175859+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173367+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:31.173367+0000","last_peered":"2024-09-15T03:20:31.173367+0000","last_clean":"2024-09-15T03:20:31.173367+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:31.173367+0000","last_undegraded":"2024-09-15T03:20:31.173367+0000","last_fullsized":"2024-09-15T03:20:31.173367+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15636,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:35.171317+0000","last_change":"2024-09-15T03:20:35.171084+0000","last_active":"2024-09-15T03:20:35.171317+0000","last_peered":"2024-09-15T03:20:35.171317+0000","last_clean":"2024-09-15T03:20:35.171317+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:35.171317+0000","last_undegraded":"2024-09-15T03:20:35.171317+0000","last_fullsized":"2024-09-15T03:20:35.171317+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:35.170789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:35.170789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:25.514810+0000","scrub_duration":0.027974141000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16174,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.183141+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:31.183141+0000","last_peered":"2024-09-15T03:20:31.183141+0000","last_clean":"2024-09-15T03:20:31.183141+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:31.183141+0000","last_undegraded":"2024-09-15T03:20:31.183141+0000","last_fullsized":"2024-09-15T03:20:31.183141+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175659+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:31.175659+0000","last_peered":"2024-09-15T03:20:31.175659+0000","last_clean":"2024-09-15T03:20:31.175659+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:31.175659+0000","last_undegraded":"2024-09-15T03:20:31.175659+0000","last_fullsized":"2024-09-15T03:20:31.175659+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16145,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173802+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:31.173802+0000","last_peered":"2024-09-15T03:20:31.173802+0000","last_clean":"2024-09-15T03:20:31.173802+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:31.173802+0000","last_undegraded":"2024-09-15T03:20:31.173802+0000","last_fullsized":"2024-09-15T03:20:31.173802+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15977,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.188829+0000","last_change":"2024-09-15T03:20:32.188544+0000","last_active":"2024-09-15T03:20:32.188829+0000","last_peered":"2024-09-15T03:20:32.188829+0000","last_clean":"2024-09-15T03:20:32.188829+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:32.188829+0000","last_undegraded":"2024-09-15T03:20:32.188829+0000","last_fullsized":"2024-09-15T03:20:32.188829+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:32.188277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.188277+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:53.497786+0000","scrub_duration":0.031810063999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16334,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175447+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:31.175447+0000","last_peered":"2024-09-15T03:20:31.175447+0000","last_clean":"2024-09-15T03:20:31.175447+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:31.175447+0000","last_undegraded":"2024-09-15T03:20:31.175447+0000","last_fullsized":"2024-09-15T03:20:31.175447+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175573+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:31.175573+0000","last_peered":"2024-09-15T03:20:31.175573+0000","last_clean":"2024-09-15T03:20:31.175573+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:31.175573+0000","last_undegraded":"2024-09-15T03:20:31.175573+0000","last_fullsized":"2024-09-15T03:20:31.175573+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12015,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175615+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:31.175615+0000","last_peered":"2024-09-15T03:20:31.175615+0000","last_clean":"2024-09-15T03:20:31.175615+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:31.175615+0000","last_undegraded":"2024-09-15T03:20:31.175615+0000","last_fullsized":"2024-09-15T03:20:31.175615+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175777+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:31.175777+0000","last_peered":"2024-09-15T03:20:31.175777+0000","last_clean":"2024-09-15T03:20:31.175777+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:31.175777+0000","last_undegraded":"2024-09-15T03:20:31.175777+0000","last_fullsized":"2024-09-15T03:20:31.175777+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932244,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439596,"statfs":{"total":96636764160,"available":95682146304,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919300,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452540,"statfs":{"total":96636764160,"available":95695400960,"internally_reserved":0,"allocated":516096,"data_stored":224741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.65900000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.57599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.64200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923912,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447928,"statfs":{"total":96636764160,"available":95690678272,"internally_reserved":0,"allocated":520192,"data_stored":225691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.501},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.746}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:37.921 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-15T03:20:37.921 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T03:20:37.931 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T03:20:37.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:20:38.230 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:38.230 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":63,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:20:32.168910+0000","last_up_change":"2024-09-15T03:06:06.823776+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:04:52.771663+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":49,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":28132},{"type":"v1","addr":"172.21.15.60:6801","nonce":28132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":28132},{"type":"v1","addr":"172.21.15.60:6803","nonce":28132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":28132},{"type":"v1","addr":"172.21.15.60:6807","nonce":28132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":28132},{"type":"v1","addr":"172.21.15.60:6805","nonce":28132}]},"public_addr":"172.21.15.60:6801/28132","cluster_addr":"172.21.15.60:6803/28132","heartbeat_back_addr":"172.21.15.60:6807/28132","heartbeat_front_addr":"172.21.15.60:6805/28132","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":60,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":32304},{"type":"v1","addr":"172.21.15.60:6817","nonce":32304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":32304},{"type":"v1","addr":"172.21.15.60:6819","nonce":32304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":32304},{"type":"v1","addr":"172.21.15.60:6823","nonce":32304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":32304},{"type":"v1","addr":"172.21.15.60:6821","nonce":32304}]},"public_addr":"172.21.15.60:6817/32304","cluster_addr":"172.21.15.60:6819/32304","heartbeat_back_addr":"172.21.15.60:6823/32304","heartbeat_front_addr":"172.21.15.60:6821/32304","state":["exists","up"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":62,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":29026},{"type":"v1","addr":"172.21.15.60:6809","nonce":29026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":29026},{"type":"v1","addr":"172.21.15.60:6811","nonce":29026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":29026},{"type":"v1","addr":"172.21.15.60:6815","nonce":29026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":29026},{"type":"v1","addr":"172.21.15.60:6813","nonce":29026}]},"public_addr":"172.21.15.60:6809/29026","cluster_addr":"172.21.15.60:6811/29026","heartbeat_back_addr":"172.21.15.60:6815/29026","heartbeat_front_addr":"172.21.15.60:6813/29026","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:20:38.245 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:20:38.246 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:38.538 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:38.552 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-09-15T03:20:37.043359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701612,"kb_used_data":2172,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785748,"statfs":{"total":386547056640,"available":382756605952,"internally_reserved":0,"allocated":2224128,"data_stored":1039932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-69690,"num_objects":-10,"num_object_clones":0,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-59525,"num_write_kb":-238136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9525,"num_bytes_recovered":-30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002365"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14924,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175859+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:31.175859+0000","last_peered":"2024-09-15T03:20:31.175859+0000","last_clean":"2024-09-15T03:20:31.175859+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:31.175859+0000","last_undegraded":"2024-09-15T03:20:31.175859+0000","last_fullsized":"2024-09-15T03:20:31.175859+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173367+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:31.173367+0000","last_peered":"2024-09-15T03:20:31.173367+0000","last_clean":"2024-09-15T03:20:31.173367+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:31.173367+0000","last_undegraded":"2024-09-15T03:20:31.173367+0000","last_fullsized":"2024-09-15T03:20:31.173367+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15636,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:35.171317+0000","last_change":"2024-09-15T03:20:35.171084+0000","last_active":"2024-09-15T03:20:35.171317+0000","last_peered":"2024-09-15T03:20:35.171317+0000","last_clean":"2024-09-15T03:20:35.171317+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:35.171317+0000","last_undegraded":"2024-09-15T03:20:35.171317+0000","last_fullsized":"2024-09-15T03:20:35.171317+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:35.170789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:35.170789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:25.514810+0000","scrub_duration":0.027974141000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16174,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.183141+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:31.183141+0000","last_peered":"2024-09-15T03:20:31.183141+0000","last_clean":"2024-09-15T03:20:31.183141+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:31.183141+0000","last_undegraded":"2024-09-15T03:20:31.183141+0000","last_fullsized":"2024-09-15T03:20:31.183141+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175659+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:31.175659+0000","last_peered":"2024-09-15T03:20:31.175659+0000","last_clean":"2024-09-15T03:20:31.175659+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:31.175659+0000","last_undegraded":"2024-09-15T03:20:31.175659+0000","last_fullsized":"2024-09-15T03:20:31.175659+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16145,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173802+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:31.173802+0000","last_peered":"2024-09-15T03:20:31.173802+0000","last_clean":"2024-09-15T03:20:31.173802+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:31.173802+0000","last_undegraded":"2024-09-15T03:20:31.173802+0000","last_fullsized":"2024-09-15T03:20:31.173802+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15977,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.188829+0000","last_change":"2024-09-15T03:20:32.188544+0000","last_active":"2024-09-15T03:20:32.188829+0000","last_peered":"2024-09-15T03:20:32.188829+0000","last_clean":"2024-09-15T03:20:32.188829+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:32.188829+0000","last_undegraded":"2024-09-15T03:20:32.188829+0000","last_fullsized":"2024-09-15T03:20:32.188829+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:32.188277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.188277+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:53.497786+0000","scrub_duration":0.031810063999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16334,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175447+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:31.175447+0000","last_peered":"2024-09-15T03:20:31.175447+0000","last_clean":"2024-09-15T03:20:31.175447+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:31.175447+0000","last_undegraded":"2024-09-15T03:20:31.175447+0000","last_fullsized":"2024-09-15T03:20:31.175447+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175573+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:31.175573+0000","last_peered":"2024-09-15T03:20:31.175573+0000","last_clean":"2024-09-15T03:20:31.175573+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:31.175573+0000","last_undegraded":"2024-09-15T03:20:31.175573+0000","last_fullsized":"2024-09-15T03:20:31.175573+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12015,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175615+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:31.175615+0000","last_peered":"2024-09-15T03:20:31.175615+0000","last_clean":"2024-09-15T03:20:31.175615+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:31.175615+0000","last_undegraded":"2024-09-15T03:20:31.175615+0000","last_fullsized":"2024-09-15T03:20:31.175615+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175777+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:31.175777+0000","last_peered":"2024-09-15T03:20:31.175777+0000","last_clean":"2024-09-15T03:20:31.175777+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:31.175777+0000","last_undegraded":"2024-09-15T03:20:31.175777+0000","last_fullsized":"2024-09-15T03:20:31.175777+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932244,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439596,"statfs":{"total":96636764160,"available":95682146304,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919300,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452540,"statfs":{"total":96636764160,"available":95695400960,"internally_reserved":0,"allocated":516096,"data_stored":224741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.65900000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.57599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.64200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923912,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447928,"statfs":{"total":96636764160,"available":95690678272,"internally_reserved":0,"allocated":520192,"data_stored":225691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.501},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.746}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:38.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:38.841 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:38.844 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:38.854 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-09-15T03:20:37.043359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701612,"kb_used_data":2172,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785748,"statfs":{"total":386547056640,"available":382756605952,"internally_reserved":0,"allocated":2224128,"data_stored":1039932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-69690,"num_objects":-10,"num_object_clones":0,"num_object_copies":-40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-59525,"num_write_kb":-238136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9525,"num_bytes_recovered":-30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002365"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14924,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175859+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:31.175859+0000","last_peered":"2024-09-15T03:20:31.175859+0000","last_clean":"2024-09-15T03:20:31.175859+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:31.175859+0000","last_undegraded":"2024-09-15T03:20:31.175859+0000","last_fullsized":"2024-09-15T03:20:31.175859+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16553,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173367+0000","last_change":"2024-09-15T03:19:56.602224+0000","last_active":"2024-09-15T03:20:31.173367+0000","last_peered":"2024-09-15T03:20:31.173367+0000","last_clean":"2024-09-15T03:20:31.173367+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:31.173367+0000","last_undegraded":"2024-09-15T03:20:31.173367+0000","last_fullsized":"2024-09-15T03:20:31.173367+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12219,"num_write_kb":48871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15636,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:35.171317+0000","last_change":"2024-09-15T03:20:35.171084+0000","last_active":"2024-09-15T03:20:35.171317+0000","last_peered":"2024-09-15T03:20:35.171317+0000","last_clean":"2024-09-15T03:20:35.171317+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:35.171317+0000","last_undegraded":"2024-09-15T03:20:35.171317+0000","last_fullsized":"2024-09-15T03:20:35.171317+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:35.170789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:35.170789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:25.514810+0000","scrub_duration":0.027974141000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16174,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.183141+0000","last_change":"2024-09-15T03:18:50.389490+0000","last_active":"2024-09-15T03:20:31.183141+0000","last_peered":"2024-09-15T03:20:31.183141+0000","last_clean":"2024-09-15T03:20:31.183141+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:31.183141+0000","last_undegraded":"2024-09-15T03:20:31.183141+0000","last_fullsized":"2024-09-15T03:20:31.183141+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11954,"num_write_kb":47820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175659+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:31.175659+0000","last_peered":"2024-09-15T03:20:31.175659+0000","last_clean":"2024-09-15T03:20:31.175659+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:31.175659+0000","last_undegraded":"2024-09-15T03:20:31.175659+0000","last_fullsized":"2024-09-15T03:20:31.175659+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16145,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173802+0000","last_change":"2024-09-15T03:19:42.560096+0000","last_active":"2024-09-15T03:20:31.173802+0000","last_peered":"2024-09-15T03:20:31.173802+0000","last_clean":"2024-09-15T03:20:31.173802+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:31.173802+0000","last_undegraded":"2024-09-15T03:20:31.173802+0000","last_fullsized":"2024-09-15T03:20:31.173802+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20907,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11877,"num_write_kb":47516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2084,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15977,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.188829+0000","last_change":"2024-09-15T03:20:32.188544+0000","last_active":"2024-09-15T03:20:32.188829+0000","last_peered":"2024-09-15T03:20:32.188829+0000","last_clean":"2024-09-15T03:20:32.188829+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:32.188829+0000","last_undegraded":"2024-09-15T03:20:32.188829+0000","last_fullsized":"2024-09-15T03:20:32.188829+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:32.188277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.188277+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:53.497786+0000","scrub_duration":0.031810063999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16334,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175447+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:31.175447+0000","last_peered":"2024-09-15T03:20:31.175447+0000","last_clean":"2024-09-15T03:20:31.175447+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:31.175447+0000","last_undegraded":"2024-09-15T03:20:31.175447+0000","last_fullsized":"2024-09-15T03:20:31.175447+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175573+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:31.175573+0000","last_peered":"2024-09-15T03:20:31.175573+0000","last_clean":"2024-09-15T03:20:31.175573+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:31.175573+0000","last_undegraded":"2024-09-15T03:20:31.175573+0000","last_fullsized":"2024-09-15T03:20:31.175573+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12015,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175615+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:31.175615+0000","last_peered":"2024-09-15T03:20:31.175615+0000","last_clean":"2024-09-15T03:20:31.175615+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:31.175615+0000","last_undegraded":"2024-09-15T03:20:31.175615+0000","last_fullsized":"2024-09-15T03:20:31.175615+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175777+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:31.175777+0000","last_peered":"2024-09-15T03:20:31.175777+0000","last_clean":"2024-09-15T03:20:31.175777+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:31.175777+0000","last_undegraded":"2024-09-15T03:20:31.175777+0000","last_fullsized":"2024-09-15T03:20:31.175777+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":125442,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131154,"num_write_kb":524675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18528,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932244,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439596,"statfs":{"total":96636764160,"available":95682146304,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919300,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452540,"statfs":{"total":96636764160,"available":95695400960,"internally_reserved":0,"allocated":516096,"data_stored":224741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.69699999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.65900000000000003}]}]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.57599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.64200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247996,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923912,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447928,"statfs":{"total":96636764160,"available":95690678272,"internally_reserved":0,"allocated":520192,"data_stored":225691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.501},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.746}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.55400000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.80800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:38.855 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:20:38.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:39.144 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:39.150 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:39.163 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":536,"stamp":"2024-09-15T03:20:39.043914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83628,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95104,"num_write_kb":380468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12284,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701616,"kb_used_data":2188,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785744,"statfs":{"total":386547056640,"available":382756601856,"internally_reserved":0,"allocated":2240512,"data_stored":1046978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-111504,"num_objects":-16,"num_object_clones":0,"num_object_copies":-64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-16,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-95575,"num_write_kb":-382343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-15769,"num_bytes_recovered":-30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002216"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14924,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175859+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:31.175859+0000","last_peered":"2024-09-15T03:20:31.175859+0000","last_clean":"2024-09-15T03:20:31.175859+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:31.175859+0000","last_undegraded":"2024-09-15T03:20:31.175859+0000","last_fullsized":"2024-09-15T03:20:31.175859+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16563,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:33.466460+0000","last_change":"2024-09-15T03:20:33.466371+0000","last_active":"2024-09-15T03:20:33.466460+0000","last_peered":"2024-09-15T03:20:33.466460+0000","last_clean":"2024-09-15T03:20:33.466460+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:33.466460+0000","last_undegraded":"2024-09-15T03:20:33.466460+0000","last_fullsized":"2024-09-15T03:20:33.466460+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12220","last_scrub_stamp":"2024-09-15T03:20:33.466246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:33.466246+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:09:32.552902+0000","scrub_duration":0.023667457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15636,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:35.171317+0000","last_change":"2024-09-15T03:20:35.171084+0000","last_active":"2024-09-15T03:20:35.171317+0000","last_peered":"2024-09-15T03:20:35.171317+0000","last_clean":"2024-09-15T03:20:35.171317+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:35.171317+0000","last_undegraded":"2024-09-15T03:20:35.171317+0000","last_fullsized":"2024-09-15T03:20:35.171317+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:35.170789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:35.170789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:25.514810+0000","scrub_duration":0.027974141000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16184,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:33.892622+0000","last_change":"2024-09-15T03:20:33.892577+0000","last_active":"2024-09-15T03:20:33.892622+0000","last_peered":"2024-09-15T03:20:33.892622+0000","last_clean":"2024-09-15T03:20:33.892622+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:33.892622+0000","last_undegraded":"2024-09-15T03:20:33.892622+0000","last_fullsized":"2024-09-15T03:20:33.892622+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11956","last_scrub_stamp":"2024-09-15T03:20:33.892476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:33.892476+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:41:18.291398+0000","scrub_duration":0.025334049000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175659+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:31.175659+0000","last_peered":"2024-09-15T03:20:31.175659+0000","last_clean":"2024-09-15T03:20:31.175659+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:31.175659+0000","last_undegraded":"2024-09-15T03:20:31.175659+0000","last_fullsized":"2024-09-15T03:20:31.175659+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16155,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:33.229541+0000","last_change":"2024-09-15T03:20:33.229484+0000","last_active":"2024-09-15T03:20:33.229541+0000","last_peered":"2024-09-15T03:20:33.229541+0000","last_clean":"2024-09-15T03:20:33.229541+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:33.229541+0000","last_undegraded":"2024-09-15T03:20:33.229541+0000","last_fullsized":"2024-09-15T03:20:33.229541+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11880","last_scrub_stamp":"2024-09-15T03:20:33.229376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:33.229376+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:52:12.669106+0000","scrub_duration":0.021090860999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15977,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.188829+0000","last_change":"2024-09-15T03:20:32.188544+0000","last_active":"2024-09-15T03:20:32.188829+0000","last_peered":"2024-09-15T03:20:32.188829+0000","last_clean":"2024-09-15T03:20:32.188829+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:32.188829+0000","last_undegraded":"2024-09-15T03:20:32.188829+0000","last_fullsized":"2024-09-15T03:20:32.188829+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:32.188277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.188277+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:53.497786+0000","scrub_duration":0.031810063999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16334,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175447+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:31.175447+0000","last_peered":"2024-09-15T03:20:31.175447+0000","last_clean":"2024-09-15T03:20:31.175447+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:31.175447+0000","last_undegraded":"2024-09-15T03:20:31.175447+0000","last_fullsized":"2024-09-15T03:20:31.175447+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175573+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:31.175573+0000","last_peered":"2024-09-15T03:20:31.175573+0000","last_clean":"2024-09-15T03:20:31.175573+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:31.175573+0000","last_undegraded":"2024-09-15T03:20:31.175573+0000","last_fullsized":"2024-09-15T03:20:31.175573+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12015,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175615+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:31.175615+0000","last_peered":"2024-09-15T03:20:31.175615+0000","last_clean":"2024-09-15T03:20:31.175615+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:31.175615+0000","last_undegraded":"2024-09-15T03:20:31.175615+0000","last_fullsized":"2024-09-15T03:20:31.175615+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175777+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:31.175777+0000","last_peered":"2024-09-15T03:20:31.175777+0000","last_clean":"2024-09-15T03:20:31.175777+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:31.175777+0000","last_undegraded":"2024-09-15T03:20:31.175777+0000","last_fullsized":"2024-09-15T03:20:31.175777+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":83628,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95104,"num_write_kb":380468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12284,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932236,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439604,"statfs":{"total":96636764160,"available":95682154496,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.73599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.83499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919308,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452532,"statfs":{"total":96636764160,"available":95695392768,"internally_reserved":0,"allocated":524288,"data_stored":228264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.628},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.76100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.70699999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.57599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.64200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923916,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447924,"statfs":{"total":96636764160,"available":95690674176,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.443},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48099999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.61899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.82299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:39.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:20:39.455 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:39.455 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":63,"fsid":"c8b63eb8-dbfb-4eae-8fe4-45d2092a1e99","created":"2024-09-15T03:04:36.945392+0000","modified":"2024-09-15T03:20:32.168910+0000","last_up_change":"2024-09-15T03:06:06.823776+0000","last_in_change":"2024-09-15T03:04:39.157850+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T03:04:52.771663+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"6156de49-3ab1-449c-a1c6-08954b367614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":49,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":28132},{"type":"v1","addr":"172.21.15.60:6801","nonce":28132}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":28132},{"type":"v1","addr":"172.21.15.60:6803","nonce":28132}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":28132},{"type":"v1","addr":"172.21.15.60:6807","nonce":28132}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":28132},{"type":"v1","addr":"172.21.15.60:6805","nonce":28132}]},"public_addr":"172.21.15.60:6801/28132","cluster_addr":"172.21.15.60:6803/28132","heartbeat_back_addr":"172.21.15.60:6807/28132","heartbeat_front_addr":"172.21.15.60:6805/28132","state":["exists","up"]},{"osd":1,"uuid":"6cddb0b6-6f15-4fb6-a24b-4741e438204b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":60,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":32304},{"type":"v1","addr":"172.21.15.60:6817","nonce":32304}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":32304},{"type":"v1","addr":"172.21.15.60:6819","nonce":32304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":32304},{"type":"v1","addr":"172.21.15.60:6823","nonce":32304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":32304},{"type":"v1","addr":"172.21.15.60:6821","nonce":32304}]},"public_addr":"172.21.15.60:6817/32304","cluster_addr":"172.21.15.60:6819/32304","heartbeat_back_addr":"172.21.15.60:6823/32304","heartbeat_front_addr":"172.21.15.60:6821/32304","state":["exists","up"]},{"osd":2,"uuid":"733d9ab1-4b03-48a7-8953-0ab0e40ef743","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":22283},{"type":"v1","addr":"172.21.15.60:6825","nonce":22283}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":22283},{"type":"v1","addr":"172.21.15.60:6827","nonce":22283}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":22283},{"type":"v1","addr":"172.21.15.60:6831","nonce":22283}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":22283},{"type":"v1","addr":"172.21.15.60:6829","nonce":22283}]},"public_addr":"172.21.15.60:6825/22283","cluster_addr":"172.21.15.60:6827/22283","heartbeat_back_addr":"172.21.15.60:6831/22283","heartbeat_front_addr":"172.21.15.60:6829/22283","state":["exists","up"]},{"osd":3,"uuid":"baf7b285-0b3b-402f-bf3b-2fc9316f8b3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":62,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":29026},{"type":"v1","addr":"172.21.15.60:6809","nonce":29026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":29026},{"type":"v1","addr":"172.21.15.60:6811","nonce":29026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":29026},{"type":"v1","addr":"172.21.15.60:6815","nonce":29026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":29026},{"type":"v1","addr":"172.21.15.60:6813","nonce":29026}]},"public_addr":"172.21.15.60:6809/29026","cluster_addr":"172.21.15.60:6811/29026","heartbeat_back_addr":"172.21.15.60:6815/29026","heartbeat_front_addr":"172.21.15.60:6813/29026","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.153978+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.369427+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.674140+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:04:44.391272+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:20:40.469 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T03:20:40.470 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:40.634 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:20:40.634 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:40.634 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:20:40.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T03:20:40.951 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T03:20:40.961 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T03:20:40.961 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:41.109 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:20:41.109 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:41.109 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:20:41.121 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T03:20:41.416 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T03:20:41.429 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T03:20:41.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:41.573 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:20:41.573 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:41.574 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:20:41.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T03:20:41.889 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T03:20:41.900 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T03:20:41.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:42.036 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-09-15T03:20:42.036 INFO:teuthology.orchestra.run.smithi060.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:42.036 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-09-15T03:20:42.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T03:20:42.363 INFO:teuthology.orchestra.run.smithi060.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T03:20:42.376 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:42.666 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:20:42.668 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:20:42.679 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2024-09-15T03:20:41.044475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83628,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95104,"num_write_kb":380468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12284,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701616,"kb_used_data":2188,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785744,"statfs":{"total":386547056640,"available":382756601856,"internally_reserved":0,"allocated":2240512,"data_stored":1046978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-111504,"num_objects":-16,"num_object_clones":0,"num_object_copies":-64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-16,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-95575,"num_write_kb":-382343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-15769,"num_bytes_recovered":-30908416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.873352"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15949,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.267876+0000","last_change":"2024-09-15T03:20:32.267876+0000","last_active":"2024-09-15T03:20:32.267876+0000","last_peered":"2024-09-15T03:20:32.267876+0000","last_clean":"2024-09-15T03:20:32.267876+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:32.267876+0000","last_undegraded":"2024-09-15T03:20:32.267876+0000","last_fullsized":"2024-09-15T03:20:32.267876+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:37:44.722078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14924,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175859+0000","last_change":"2024-09-15T03:17:46.614763+0000","last_active":"2024-09-15T03:20:31.175859+0000","last_peered":"2024-09-15T03:20:31.175859+0000","last_clean":"2024-09-15T03:20:31.175859+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:31.175859+0000","last_undegraded":"2024-09-15T03:20:31.175859+0000","last_fullsized":"2024-09-15T03:20:31.175859+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":45700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2002,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16563,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:33.466460+0000","last_change":"2024-09-15T03:20:33.466371+0000","last_active":"2024-09-15T03:20:33.466460+0000","last_peered":"2024-09-15T03:20:33.466460+0000","last_clean":"2024-09-15T03:20:33.466460+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:33.466460+0000","last_undegraded":"2024-09-15T03:20:33.466460+0000","last_fullsized":"2024-09-15T03:20:33.466460+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12220","last_scrub_stamp":"2024-09-15T03:20:33.466246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:33.466246+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:09:32.552902+0000","scrub_duration":0.023667457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15636,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:35.171317+0000","last_change":"2024-09-15T03:20:35.171084+0000","last_active":"2024-09-15T03:20:35.171317+0000","last_peered":"2024-09-15T03:20:35.171317+0000","last_clean":"2024-09-15T03:20:35.171317+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:35.171317+0000","last_undegraded":"2024-09-15T03:20:35.171317+0000","last_fullsized":"2024-09-15T03:20:35.171317+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:35.170789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:35.170789+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:34:25.514810+0000","scrub_duration":0.027974141000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16408,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.236458+0000","last_change":"2024-09-15T03:20:32.236400+0000","last_active":"2024-09-15T03:20:32.236458+0000","last_peered":"2024-09-15T03:20:32.236458+0000","last_clean":"2024-09-15T03:20:32.236458+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:32.236458+0000","last_undegraded":"2024-09-15T03:20:32.236458+0000","last_fullsized":"2024-09-15T03:20:32.236458+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:32.236272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.236272+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:02:24.043337+0000","scrub_duration":0.022354303999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16199,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.173503+0000","last_change":"2024-09-15T03:20:08.644599+0000","last_active":"2024-09-15T03:20:31.173503+0000","last_peered":"2024-09-15T03:20:31.173503+0000","last_clean":"2024-09-15T03:20:31.173503+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:31.173503+0000","last_undegraded":"2024-09-15T03:20:31.173503+0000","last_fullsized":"2024-09-15T03:20:31.173503+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:53:38.130074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16184,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:33.892622+0000","last_change":"2024-09-15T03:20:33.892577+0000","last_active":"2024-09-15T03:20:33.892622+0000","last_peered":"2024-09-15T03:20:33.892622+0000","last_clean":"2024-09-15T03:20:33.892622+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:33.892622+0000","last_undegraded":"2024-09-15T03:20:33.892622+0000","last_fullsized":"2024-09-15T03:20:33.892622+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11956","last_scrub_stamp":"2024-09-15T03:20:33.892476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:33.892476+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:41:18.291398+0000","scrub_duration":0.025334049000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12237,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175659+0000","last_change":"2024-09-15T03:18:19.268917+0000","last_active":"2024-09-15T03:20:31.175659+0000","last_peered":"2024-09-15T03:20:31.175659+0000","last_clean":"2024-09-15T03:20:31.175659+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:31.175659+0000","last_undegraded":"2024-09-15T03:20:31.175659+0000","last_fullsized":"2024-09-15T03:20:31.175659+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12081,"num_write_kb":48332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16168,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.252637+0000","last_change":"2024-09-15T03:20:31.252493+0000","last_active":"2024-09-15T03:20:31.252637+0000","last_peered":"2024-09-15T03:20:31.252637+0000","last_clean":"2024-09-15T03:20:31.252637+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:31.252637+0000","last_undegraded":"2024-09-15T03:20:31.252637+0000","last_fullsized":"2024-09-15T03:20:31.252637+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.252365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.252365+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:22.469638+0000","scrub_duration":0.026928457999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16224,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.401096+0000","last_change":"2024-09-15T03:20:31.401010+0000","last_active":"2024-09-15T03:20:31.401096+0000","last_peered":"2024-09-15T03:20:31.401096+0000","last_clean":"2024-09-15T03:20:31.401096+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:31.401096+0000","last_undegraded":"2024-09-15T03:20:31.401096+0000","last_fullsized":"2024-09-15T03:20:31.401096+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:31.400901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:31.400901+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:45:34.465530+0000","scrub_duration":0.025277325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16155,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:33.229541+0000","last_change":"2024-09-15T03:20:33.229484+0000","last_active":"2024-09-15T03:20:33.229541+0000","last_peered":"2024-09-15T03:20:33.229541+0000","last_clean":"2024-09-15T03:20:33.229541+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:33.229541+0000","last_undegraded":"2024-09-15T03:20:33.229541+0000","last_fullsized":"2024-09-15T03:20:33.229541+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11880","last_scrub_stamp":"2024-09-15T03:20:33.229376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:33.229376+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:52:12.669106+0000","scrub_duration":0.021090860999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15977,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:32.188829+0000","last_change":"2024-09-15T03:20:32.188544+0000","last_active":"2024-09-15T03:20:32.188829+0000","last_peered":"2024-09-15T03:20:32.188829+0000","last_clean":"2024-09-15T03:20:32.188829+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:32.188829+0000","last_undegraded":"2024-09-15T03:20:32.188829+0000","last_fullsized":"2024-09-15T03:20:32.188829+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:32.188277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:32.188277+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:08:53.497786+0000","scrub_duration":0.031810063999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16334,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175447+0000","last_change":"2024-09-15T03:15:01.389034+0000","last_active":"2024-09-15T03:20:31.175447+0000","last_peered":"2024-09-15T03:20:31.175447+0000","last_clean":"2024-09-15T03:20:31.175447+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:31.175447+0000","last_undegraded":"2024-09-15T03:20:31.175447+0000","last_fullsized":"2024-09-15T03:20:31.175447+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12131,"num_write_kb":48527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2041,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11799,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175573+0000","last_change":"2024-09-15T03:20:20.267276+0000","last_active":"2024-09-15T03:20:31.175573+0000","last_peered":"2024-09-15T03:20:31.175573+0000","last_clean":"2024-09-15T03:20:31.175573+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:31.175573+0000","last_undegraded":"2024-09-15T03:20:31.175573+0000","last_fullsized":"2024-09-15T03:20:31.175573+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":46724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12015,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175615+0000","last_change":"2024-09-15T03:19:27.504838+0000","last_active":"2024-09-15T03:20:31.175615+0000","last_peered":"2024-09-15T03:20:31.175615+0000","last_clean":"2024-09-15T03:20:31.175615+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:31.175615+0000","last_undegraded":"2024-09-15T03:20:31.175615+0000","last_fullsized":"2024-09-15T03:20:31.175615+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11879,"num_write_kb":47525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12149,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T03:20:31.175777+0000","last_change":"2024-09-15T03:19:15.457715+0000","last_active":"2024-09-15T03:20:31.175777+0000","last_peered":"2024-09-15T03:20:31.175777+0000","last_clean":"2024-09-15T03:20:31.175777+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:31.175777+0000","last_undegraded":"2024-09-15T03:20:31.175777+0000","last_fullsized":"2024-09-15T03:20:31.175777+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:04:53.732736+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13938,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":48028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2086,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":83628,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95104,"num_write_kb":380468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12284,"num_bytes_recovered":163840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932236,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439604,"statfs":{"total":96636764160,"available":95682154496,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Sun Sep 15 03:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":3.5910000000000002,"15min":2.8730000000000002},"min":{"1min":0.36599999999999999,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":2.423,"5min":35.402000000000001,"15min":40.710999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":2.137,"15min":1.944},"min":{"1min":0.40200000000000002,"5min":0.22700000000000001,"15min":0.20300000000000001},"max":{"1min":2.4849999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.73599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.83499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919308,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452532,"statfs":{"total":96636764160,"available":95695392768,"internally_reserved":0,"allocated":524288,"data_stored":228264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.84799999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.628},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.76100000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.70699999999999996}]}]},{"osd":1,"up_from":29,"seq":124554051757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.33300000000000002}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.57599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.64200000000000002}]}]},{"osd":0,"up_from":23,"seq":98784247997,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923916,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447924,"statfs":{"total":96636764160,"available":95690674176,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.9970000000000001,"15min":1.6919999999999999},"min":{"1min":0.22500000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.76700000000000002,"5min":22.384,"15min":22.384},"last":0.443},{"interface":"front","average":{"1min":0.56599999999999995,"5min":2.8279999999999998,"15min":2.585},"min":{"1min":0.39600000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.79700000000000004,"5min":30.433,"15min":30.433},"last":0.48099999999999998}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":2.0670000000000002,"15min":2.323},"min":{"1min":0.30499999999999999,"5min":0.221,"15min":0.16200000000000001},"max":{"1min":0.64700000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.71,"15min":1.728},"min":{"1min":0.40500000000000003,"5min":0.23100000000000001,"15min":0.21199999999999999},"max":{"1min":0.81799999999999995,"5min":12.281000000000001,"15min":20.387},"last":0.61899999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.82299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:42.683 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-09-15T03:04:53.732736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=4, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.683 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-09-15T03:04:53.732736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=4, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.684 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-09-15T03:20:33.466246+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.684 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T03:20:35.170789+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=35, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.684 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-15T03:20:32.236272+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.684 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-09-15T03:04:53.732736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=4, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.684 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-15T03:20:33.892476+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.684 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-15T03:04:53.732736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=4, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.684 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-15T03:20:31.252365+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.685 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-15T03:20:31.400901+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=31, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.685 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-15T03:20:33.229376+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=33, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.685 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-15T03:20:32.188277+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=32, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.685 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-09-15T03:04:53.732736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=4, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.685 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-09-15T03:04:53.732736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=4, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.685 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-09-15T03:04:53.732736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=4, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.685 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-09-15T03:04:53.732736+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=4, tm_sec=53, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:42.685 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T03:21:02.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:21:02.985 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:21:02.988 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:21:02.998 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":547,"stamp":"2024-09-15T03:21:01.050176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23908,"num_write_kb":95632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701612,"kb_used_data":2188,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785748,"statfs":{"total":386547056640,"available":382756605952,"internally_reserved":0,"allocated":2240512,"data_stored":1046978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-41814,"num_objects":-6,"num_object_clones":0,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-35562,"num_write_kb":-142277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003415"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15960,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:50.157507+0000","last_change":"2024-09-15T03:20:50.157454+0000","last_active":"2024-09-15T03:20:50.157507+0000","last_peered":"2024-09-15T03:20:50.157507+0000","last_clean":"2024-09-15T03:20:50.157507+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:50.157507+0000","last_undegraded":"2024-09-15T03:20:50.157507+0000","last_fullsized":"2024-09-15T03:20:50.157507+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11910","last_scrub_stamp":"2024-09-15T03:20:50.157337+0000","last_deep_scrub":"37'11910","last_deep_scrub_stamp":"2024-09-15T03:20:50.157337+0000","last_clean_scrub_stamp":"2024-09-15T03:20:50.157337+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:46:36.960808+0000","scrub_duration":0.022153322999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14935,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:38.186156+0000","last_change":"2024-09-15T03:20:38.186044+0000","last_active":"2024-09-15T03:20:38.186156+0000","last_peered":"2024-09-15T03:20:38.186156+0000","last_clean":"2024-09-15T03:20:38.186156+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:20:38.186156+0000","last_undegraded":"2024-09-15T03:20:38.186156+0000","last_fullsized":"2024-09-15T03:20:38.186156+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11424","last_scrub_stamp":"2024-09-15T03:20:38.185865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:38.185865+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.025130474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16574,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:44.582412+0000","last_change":"2024-09-15T03:20:44.582334+0000","last_active":"2024-09-15T03:20:44.582412+0000","last_peered":"2024-09-15T03:20:44.582412+0000","last_clean":"2024-09-15T03:20:44.582412+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:44.582412+0000","last_undegraded":"2024-09-15T03:20:44.582412+0000","last_fullsized":"2024-09-15T03:20:44.582412+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12220","last_scrub_stamp":"2024-09-15T03:20:44.582217+0000","last_deep_scrub":"37'12220","last_deep_scrub_stamp":"2024-09-15T03:20:44.582217+0000","last_clean_scrub_stamp":"2024-09-15T03:20:44.582217+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:32:46.864637+0000","scrub_duration":0.025945809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15647,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.240829+0000","last_change":"2024-09-15T03:20:54.240760+0000","last_active":"2024-09-15T03:20:54.240829+0000","last_peered":"2024-09-15T03:20:54.240829+0000","last_clean":"2024-09-15T03:20:54.240829+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:54.240829+0000","last_undegraded":"2024-09-15T03:20:54.240829+0000","last_fullsized":"2024-09-15T03:20:54.240829+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:54.240645+0000","last_deep_scrub":"37'11626","last_deep_scrub_stamp":"2024-09-15T03:20:54.240645+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.240645+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:28:43.029005+0000","scrub_duration":0.030165904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16419,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:46.173754+0000","last_change":"2024-09-15T03:20:46.173699+0000","last_active":"2024-09-15T03:20:46.173754+0000","last_peered":"2024-09-15T03:20:46.173754+0000","last_clean":"2024-09-15T03:20:46.173754+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:46.173754+0000","last_undegraded":"2024-09-15T03:20:46.173754+0000","last_fullsized":"2024-09-15T03:20:46.173754+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:46.173583+0000","last_deep_scrub":"37'12079","last_deep_scrub_stamp":"2024-09-15T03:20:46.173583+0000","last_clean_scrub_stamp":"2024-09-15T03:20:46.173583+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:56:07.494231+0000","scrub_duration":0.028246734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:53.120065+0000","last_change":"2024-09-15T03:20:53.119953+0000","last_active":"2024-09-15T03:20:53.120065+0000","last_peered":"2024-09-15T03:20:53.120065+0000","last_clean":"2024-09-15T03:20:53.120065+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:53.120065+0000","last_undegraded":"2024-09-15T03:20:53.120065+0000","last_fullsized":"2024-09-15T03:20:53.120065+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11998","last_scrub_stamp":"2024-09-15T03:20:53.119821+0000","last_deep_scrub":"37'11998","last_deep_scrub_stamp":"2024-09-15T03:20:53.119821+0000","last_clean_scrub_stamp":"2024-09-15T03:20:53.119821+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:29:58.610225+0000","scrub_duration":0.024877759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16195,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:41.750378+0000","last_change":"2024-09-15T03:20:41.750286+0000","last_active":"2024-09-15T03:20:41.750378+0000","last_peered":"2024-09-15T03:20:41.750378+0000","last_clean":"2024-09-15T03:20:41.750378+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:41.750378+0000","last_undegraded":"2024-09-15T03:20:41.750378+0000","last_fullsized":"2024-09-15T03:20:41.750378+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11956","last_scrub_stamp":"2024-09-15T03:20:41.750168+0000","last_deep_scrub":"37'11956","last_deep_scrub_stamp":"2024-09-15T03:20:41.750168+0000","last_clean_scrub_stamp":"2024-09-15T03:20:41.750168+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:39:39.261034+0000","scrub_duration":0.024888598000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:45.196656+0000","last_change":"2024-09-15T03:20:45.196566+0000","last_active":"2024-09-15T03:20:45.196656+0000","last_peered":"2024-09-15T03:20:45.196656+0000","last_clean":"2024-09-15T03:20:45.196656+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:45.196656+0000","last_undegraded":"2024-09-15T03:20:45.196656+0000","last_fullsized":"2024-09-15T03:20:45.196656+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12083","last_scrub_stamp":"2024-09-15T03:20:45.196432+0000","last_deep_scrub":"37'12083","last_deep_scrub_stamp":"2024-09-15T03:20:45.196432+0000","last_clean_scrub_stamp":"2024-09-15T03:20:45.196432+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:07:31.898240+0000","scrub_duration":0.026534512999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16179,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:44.220015+0000","last_change":"2024-09-15T03:20:44.219956+0000","last_active":"2024-09-15T03:20:44.220015+0000","last_peered":"2024-09-15T03:20:44.220015+0000","last_clean":"2024-09-15T03:20:44.220015+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:44.220015+0000","last_undegraded":"2024-09-15T03:20:44.220015+0000","last_fullsized":"2024-09-15T03:20:44.220015+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:44.219803+0000","last_deep_scrub":"37'11966","last_deep_scrub_stamp":"2024-09-15T03:20:44.219803+0000","last_clean_scrub_stamp":"2024-09-15T03:20:44.219803+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:53:52.192742+0000","scrub_duration":0.029574573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16235,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:43.603027+0000","last_change":"2024-09-15T03:20:43.602975+0000","last_active":"2024-09-15T03:20:43.603027+0000","last_peered":"2024-09-15T03:20:43.603027+0000","last_clean":"2024-09-15T03:20:43.603027+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:43.603027+0000","last_undegraded":"2024-09-15T03:20:43.603027+0000","last_fullsized":"2024-09-15T03:20:43.603027+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:43.602854+0000","last_deep_scrub":"37'11966","last_deep_scrub_stamp":"2024-09-15T03:20:43.602854+0000","last_clean_scrub_stamp":"2024-09-15T03:20:43.602854+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:39:16.457791+0000","scrub_duration":0.025868749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16166,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.145248+0000","last_change":"2024-09-15T03:20:51.145037+0000","last_active":"2024-09-15T03:20:51.145248+0000","last_peered":"2024-09-15T03:20:51.145248+0000","last_clean":"2024-09-15T03:20:51.145248+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:51.145248+0000","last_undegraded":"2024-09-15T03:20:51.145248+0000","last_fullsized":"2024-09-15T03:20:51.145248+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11880","last_scrub_stamp":"2024-09-15T03:20:51.144651+0000","last_deep_scrub":"37'11880","last_deep_scrub_stamp":"2024-09-15T03:20:51.144651+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.144651+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:27:36.985865+0000","scrub_duration":0.030732450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:55.227237+0000","last_change":"2024-09-15T03:20:55.227166+0000","last_active":"2024-09-15T03:20:55.227237+0000","last_peered":"2024-09-15T03:20:55.227237+0000","last_clean":"2024-09-15T03:20:55.227237+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:55.227237+0000","last_undegraded":"2024-09-15T03:20:55.227237+0000","last_fullsized":"2024-09-15T03:20:55.227237+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:55.227039+0000","last_deep_scrub":"37'11898","last_deep_scrub_stamp":"2024-09-15T03:20:55.227039+0000","last_clean_scrub_stamp":"2024-09-15T03:20:55.227039+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:40:04.144477+0000","scrub_duration":0.029692864999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16345,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:37.168706+0000","last_change":"2024-09-15T03:20:37.168545+0000","last_active":"2024-09-15T03:20:37.168706+0000","last_peered":"2024-09-15T03:20:37.168706+0000","last_clean":"2024-09-15T03:20:37.168706+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:37.168706+0000","last_undegraded":"2024-09-15T03:20:37.168706+0000","last_fullsized":"2024-09-15T03:20:37.168706+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12132","last_scrub_stamp":"2024-09-15T03:20:37.168293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:04:53.732736+0000","last_clean_scrub_stamp":"2024-09-15T03:20:37.168293+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.020632517999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11810,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:47.226096+0000","last_change":"2024-09-15T03:20:47.226017+0000","last_active":"2024-09-15T03:20:47.226096+0000","last_peered":"2024-09-15T03:20:47.226096+0000","last_clean":"2024-09-15T03:20:47.226096+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:47.226096+0000","last_undegraded":"2024-09-15T03:20:47.226096+0000","last_fullsized":"2024-09-15T03:20:47.226096+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11680","last_scrub_stamp":"2024-09-15T03:20:47.225892+0000","last_deep_scrub":"37'11680","last_deep_scrub_stamp":"2024-09-15T03:20:47.225892+0000","last_clean_scrub_stamp":"2024-09-15T03:20:47.225892+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:39:25.186283+0000","scrub_duration":0.023339927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12026,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:50.284524+0000","last_change":"2024-09-15T03:20:50.284424+0000","last_active":"2024-09-15T03:20:50.284524+0000","last_peered":"2024-09-15T03:20:50.284524+0000","last_clean":"2024-09-15T03:20:50.284524+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:50.284524+0000","last_undegraded":"2024-09-15T03:20:50.284524+0000","last_fullsized":"2024-09-15T03:20:50.284524+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11881","last_scrub_stamp":"2024-09-15T03:20:50.284272+0000","last_deep_scrub":"37'11881","last_deep_scrub_stamp":"2024-09-15T03:20:50.284272+0000","last_clean_scrub_stamp":"2024-09-15T03:20:50.284272+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:00:04.015047+0000","scrub_duration":0.024328300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12160,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:49.254331+0000","last_change":"2024-09-15T03:20:49.254253+0000","last_active":"2024-09-15T03:20:49.254331+0000","last_peered":"2024-09-15T03:20:49.254331+0000","last_clean":"2024-09-15T03:20:49.254331+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:49.254331+0000","last_undegraded":"2024-09-15T03:20:49.254331+0000","last_fullsized":"2024-09-15T03:20:49.254331+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12007","last_scrub_stamp":"2024-09-15T03:20:49.254118+0000","last_deep_scrub":"37'12007","last_deep_scrub_stamp":"2024-09-15T03:20:49.254118+0000","last_clean_scrub_stamp":"2024-09-15T03:20:49.254118+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:13:56.947211+0000","scrub_duration":0.023478103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23908,"num_write_kb":95632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932236,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439604,"statfs":{"total":96636764160,"available":95682154496,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":1.3340000000000001,"15min":1.585},"min":{"1min":0.23200000000000001,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.71099999999999997,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":1.516,"15min":1.8620000000000001},"min":{"1min":0.26600000000000001,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.72599999999999998,"5min":11.006,"15min":26.143000000000001},"last":0.58099999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":2.7879999999999998,"15min":2.8029999999999999},"min":{"1min":0.39200000000000002,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":0.88800000000000001,"5min":27.18,"15min":40.710999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":2,"15min":1.9099999999999999},"min":{"1min":0.40100000000000002,"5min":0.247,"15min":0.20300000000000001},"max":{"1min":0.85999999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.752}]},{"osd":2,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":2.0670000000000002,"15min":2.5859999999999999},"min":{"1min":0.39300000000000002,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.873,"5min":13.894,"15min":40.784999999999997},"last":0.621},{"interface":"front","average":{"1min":0.63300000000000001,"5min":1.2050000000000001,"15min":1.752},"min":{"1min":0.33300000000000002,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.94199999999999995,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.66000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919304,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452536,"statfs":{"total":96636764160,"available":95695396864,"internally_reserved":0,"allocated":524288,"data_stored":228264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":2.032,"15min":2.1739999999999999},"min":{"1min":0.29399999999999998,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.58399999999999996,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.52900000000000003,"5min":1.0940000000000001,"15min":1.4399999999999999},"min":{"1min":0.34000000000000002,"5min":0.20200000000000001,"15min":0.192},"max":{"1min":0.878,"5min":19.189,"15min":19.329000000000001},"last":0.65000000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.75900000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.6180000000000001,"15min":1.819},"min":{"1min":0.40600000000000003,"5min":0.24099999999999999,"15min":0.216},"max":{"1min":0.75700000000000001,"5min":18.731000000000002,"15min":21.600999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":1.79,"15min":1.925},"min":{"1min":0.42099999999999999,"5min":0.255,"15min":0.22700000000000001},"max":{"1min":0.90500000000000003,"5min":22.367999999999999,"15min":22.710000000000001},"last":0.85599999999999998}]}]},{"osd":1,"up_from":29,"seq":124554051761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.78400000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.74199999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.83399999999999996}]}]},{"osd":0,"up_from":23,"seq":98784248001,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923916,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447924,"statfs":{"total":96636764160,"available":95690674176,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.357,"15min":1.673},"min":{"1min":0.25600000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.67000000000000004,"5min":22.384,"15min":22.384},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.0590000000000002,"15min":2.5209999999999999},"min":{"1min":0.34599999999999997,"5min":0.253,"15min":0.19900000000000001},"max":{"1min":0.746,"5min":30.433,"15min":30.433},"last":0.73799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.6519999999999999,"15min":2.2989999999999999},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.16200000000000001},"max":{"1min":0.77300000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.62,"5min":1.181,"15min":1.7290000000000001},"min":{"1min":0.39500000000000002,"5min":0.23999999999999999,"15min":0.21199999999999999},"max":{"1min":0.82899999999999996,"5min":10.321999999999999,"15min":20.387},"last":0.82399999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.995,"15min":1.395},"min":{"1min":0.44800000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.876,"5min":15.114000000000001,"15min":20.948},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":1.2090000000000001,"15min":1.615},"min":{"1min":0.41099999999999998,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":0.91700000000000004,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:21:02.999 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-09-15T03:20:38.185865+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:03.000 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-09-15T03:20:37.168293+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=37, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=39, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:03.000 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T03:21:23.001 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:21:23.302 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-15T03:21:23.305 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-15T03:21:23.315 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":558,"stamp":"2024-09-15T03:21:23.056123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23908,"num_write_kb":95632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":3701612,"kb_used_data":2188,"kb_used_omap":14,"kb_used_meta":3699377,"kb_avail":373785748,"statfs":{"total":386547056640,"available":382756605952,"internally_reserved":0,"allocated":2240512,"data_stored":1046978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15333,"internal_metadata":3788162075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003110"},"pg_stats":[{"pgid":"1.f","version":"37'11910","reported_seq":15960,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:50.157507+0000","last_change":"2024-09-15T03:20:50.157454+0000","last_active":"2024-09-15T03:20:50.157507+0000","last_peered":"2024-09-15T03:20:50.157507+0000","last_clean":"2024-09-15T03:20:50.157507+0000","last_became_active":"2024-09-15T03:20:32.188399+0000","last_became_peered":"2024-09-15T03:20:32.188399+0000","last_unstale":"2024-09-15T03:20:50.157507+0000","last_undegraded":"2024-09-15T03:20:50.157507+0000","last_fullsized":"2024-09-15T03:20:50.157507+0000","mapping_epoch":62,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11910","last_scrub_stamp":"2024-09-15T03:20:50.157337+0000","last_deep_scrub":"37'11910","last_deep_scrub_stamp":"2024-09-15T03:20:50.157337+0000","last_clean_scrub_stamp":"2024-09-15T03:20:50.157337+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:46:36.960808+0000","scrub_duration":0.022153322999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1976,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11424","reported_seq":14945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:21:02.326484+0000","last_change":"2024-09-15T03:21:02.326395+0000","last_active":"2024-09-15T03:21:02.326484+0000","last_peered":"2024-09-15T03:21:02.326484+0000","last_clean":"2024-09-15T03:21:02.326484+0000","last_became_active":"2024-09-15T03:17:46.533065+0000","last_became_peered":"2024-09-15T03:17:46.533065+0000","last_unstale":"2024-09-15T03:21:02.326484+0000","last_undegraded":"2024-09-15T03:21:02.326484+0000","last_fullsized":"2024-09-15T03:21:02.326484+0000","mapping_epoch":38,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11424","last_scrub_stamp":"2024-09-15T03:21:02.326272+0000","last_deep_scrub":"37'11424","last_deep_scrub_stamp":"2024-09-15T03:21:02.326272+0000","last_clean_scrub_stamp":"2024-09-15T03:21:02.326272+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:06:12.202855+0000","scrub_duration":0.022351862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12220","reported_seq":16574,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:44.582412+0000","last_change":"2024-09-15T03:20:44.582334+0000","last_active":"2024-09-15T03:20:44.582412+0000","last_peered":"2024-09-15T03:20:44.582412+0000","last_clean":"2024-09-15T03:20:44.582412+0000","last_became_active":"2024-09-15T03:19:56.511453+0000","last_became_peered":"2024-09-15T03:19:56.511453+0000","last_unstale":"2024-09-15T03:20:44.582412+0000","last_undegraded":"2024-09-15T03:20:44.582412+0000","last_fullsized":"2024-09-15T03:20:44.582412+0000","mapping_epoch":56,"log_start":"31'2200","ondisk_log_start":"31'2200","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12220","last_scrub_stamp":"2024-09-15T03:20:44.582217+0000","last_deep_scrub":"37'12220","last_deep_scrub_stamp":"2024-09-15T03:20:44.582217+0000","last_clean_scrub_stamp":"2024-09-15T03:20:44.582217+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:32:46.864637+0000","scrub_duration":0.025945809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11626","reported_seq":15647,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.240829+0000","last_change":"2024-09-15T03:20:54.240760+0000","last_active":"2024-09-15T03:20:54.240829+0000","last_peered":"2024-09-15T03:20:54.240829+0000","last_clean":"2024-09-15T03:20:54.240829+0000","last_became_active":"2024-09-15T03:12:00.191581+0000","last_became_peered":"2024-09-15T03:12:00.191581+0000","last_unstale":"2024-09-15T03:20:54.240829+0000","last_undegraded":"2024-09-15T03:20:54.240829+0000","last_fullsized":"2024-09-15T03:20:54.240829+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11626","last_scrub_stamp":"2024-09-15T03:20:54.240645+0000","last_deep_scrub":"37'11626","last_deep_scrub_stamp":"2024-09-15T03:20:54.240645+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.240645+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:28:43.029005+0000","scrub_duration":0.030165904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12079","reported_seq":16419,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:46.173754+0000","last_change":"2024-09-15T03:20:46.173699+0000","last_active":"2024-09-15T03:20:46.173754+0000","last_peered":"2024-09-15T03:20:46.173754+0000","last_clean":"2024-09-15T03:20:46.173754+0000","last_became_active":"2024-09-15T03:19:02.848119+0000","last_became_peered":"2024-09-15T03:19:02.848119+0000","last_unstale":"2024-09-15T03:20:46.173754+0000","last_undegraded":"2024-09-15T03:20:46.173754+0000","last_fullsized":"2024-09-15T03:20:46.173754+0000","mapping_epoch":48,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12079","last_scrub_stamp":"2024-09-15T03:20:46.173583+0000","last_deep_scrub":"37'12079","last_deep_scrub_stamp":"2024-09-15T03:20:46.173583+0000","last_clean_scrub_stamp":"2024-09-15T03:20:46.173583+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:56:07.494231+0000","scrub_duration":0.028246734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11998","reported_seq":16210,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:53.120065+0000","last_change":"2024-09-15T03:20:53.119953+0000","last_active":"2024-09-15T03:20:53.120065+0000","last_peered":"2024-09-15T03:20:53.120065+0000","last_clean":"2024-09-15T03:20:53.120065+0000","last_became_active":"2024-09-15T03:20:08.556409+0000","last_became_peered":"2024-09-15T03:20:08.556409+0000","last_unstale":"2024-09-15T03:20:53.120065+0000","last_undegraded":"2024-09-15T03:20:53.120065+0000","last_fullsized":"2024-09-15T03:20:53.120065+0000","mapping_epoch":58,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11998","last_scrub_stamp":"2024-09-15T03:20:53.119821+0000","last_deep_scrub":"37'11998","last_deep_scrub_stamp":"2024-09-15T03:20:53.119821+0000","last_clean_scrub_stamp":"2024-09-15T03:20:53.119821+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:29:58.610225+0000","scrub_duration":0.024877759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11998,"num_write_kb":47992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11956","reported_seq":16195,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:41.750378+0000","last_change":"2024-09-15T03:20:41.750286+0000","last_active":"2024-09-15T03:20:41.750378+0000","last_peered":"2024-09-15T03:20:41.750378+0000","last_clean":"2024-09-15T03:20:41.750378+0000","last_became_active":"2024-09-15T03:18:50.299352+0000","last_became_peered":"2024-09-15T03:18:50.299352+0000","last_unstale":"2024-09-15T03:20:41.750378+0000","last_undegraded":"2024-09-15T03:20:41.750378+0000","last_fullsized":"2024-09-15T03:20:41.750378+0000","mapping_epoch":46,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11956","last_scrub_stamp":"2024-09-15T03:20:41.750168+0000","last_deep_scrub":"37'11956","last_deep_scrub_stamp":"2024-09-15T03:20:41.750168+0000","last_clean_scrub_stamp":"2024-09-15T03:20:41.750168+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:39:39.261034+0000","scrub_duration":0.024888598000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12083","reported_seq":12258,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:45.196656+0000","last_change":"2024-09-15T03:20:45.196566+0000","last_active":"2024-09-15T03:20:45.196656+0000","last_peered":"2024-09-15T03:20:45.196656+0000","last_clean":"2024-09-15T03:20:45.196656+0000","last_became_active":"2024-09-15T03:18:19.181336+0000","last_became_peered":"2024-09-15T03:18:19.181336+0000","last_unstale":"2024-09-15T03:20:45.196656+0000","last_undegraded":"2024-09-15T03:20:45.196656+0000","last_fullsized":"2024-09-15T03:20:45.196656+0000","mapping_epoch":42,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12083","last_scrub_stamp":"2024-09-15T03:20:45.196432+0000","last_deep_scrub":"37'12083","last_deep_scrub_stamp":"2024-09-15T03:20:45.196432+0000","last_clean_scrub_stamp":"2024-09-15T03:20:45.196432+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:07:31.898240+0000","scrub_duration":0.026534512999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'11966","reported_seq":16179,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:44.220015+0000","last_change":"2024-09-15T03:20:44.219956+0000","last_active":"2024-09-15T03:20:44.220015+0000","last_peered":"2024-09-15T03:20:44.220015+0000","last_clean":"2024-09-15T03:20:44.220015+0000","last_became_active":"2024-09-15T03:18:07.126771+0000","last_became_peered":"2024-09-15T03:18:07.126771+0000","last_unstale":"2024-09-15T03:20:44.220015+0000","last_undegraded":"2024-09-15T03:20:44.220015+0000","last_fullsized":"2024-09-15T03:20:44.220015+0000","mapping_epoch":40,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:44.219803+0000","last_deep_scrub":"37'11966","last_deep_scrub_stamp":"2024-09-15T03:20:44.219803+0000","last_clean_scrub_stamp":"2024-09-15T03:20:44.219803+0000","objects_scrubbed":1,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:53:52.192742+0000","scrub_duration":0.029574573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11966","reported_seq":16235,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:43.603027+0000","last_change":"2024-09-15T03:20:43.602975+0000","last_active":"2024-09-15T03:20:43.603027+0000","last_peered":"2024-09-15T03:20:43.603027+0000","last_clean":"2024-09-15T03:20:43.603027+0000","last_became_active":"2024-09-15T03:18:37.251131+0000","last_became_peered":"2024-09-15T03:18:37.251131+0000","last_unstale":"2024-09-15T03:20:43.603027+0000","last_undegraded":"2024-09-15T03:20:43.603027+0000","last_fullsized":"2024-09-15T03:20:43.603027+0000","mapping_epoch":44,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11966","last_scrub_stamp":"2024-09-15T03:20:43.602854+0000","last_deep_scrub":"37'11966","last_deep_scrub_stamp":"2024-09-15T03:20:43.602854+0000","last_clean_scrub_stamp":"2024-09-15T03:20:43.602854+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:39:16.457791+0000","scrub_duration":0.025868749999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11880","reported_seq":16166,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.145248+0000","last_change":"2024-09-15T03:20:51.145037+0000","last_active":"2024-09-15T03:20:51.145248+0000","last_peered":"2024-09-15T03:20:51.145248+0000","last_clean":"2024-09-15T03:20:51.145248+0000","last_became_active":"2024-09-15T03:19:42.461020+0000","last_became_peered":"2024-09-15T03:19:42.461020+0000","last_unstale":"2024-09-15T03:20:51.145248+0000","last_undegraded":"2024-09-15T03:20:51.145248+0000","last_fullsized":"2024-09-15T03:20:51.145248+0000","mapping_epoch":54,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11880","last_scrub_stamp":"2024-09-15T03:20:51.144651+0000","last_deep_scrub":"37'11880","last_deep_scrub_stamp":"2024-09-15T03:20:51.144651+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.144651+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:27:36.985865+0000","scrub_duration":0.030732450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11898","reported_seq":15988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:55.227237+0000","last_change":"2024-09-15T03:20:55.227166+0000","last_active":"2024-09-15T03:20:55.227237+0000","last_peered":"2024-09-15T03:20:55.227237+0000","last_clean":"2024-09-15T03:20:55.227237+0000","last_became_active":"2024-09-15T03:09:41.980811+0000","last_became_peered":"2024-09-15T03:09:41.980811+0000","last_unstale":"2024-09-15T03:20:55.227237+0000","last_undegraded":"2024-09-15T03:20:55.227237+0000","last_fullsized":"2024-09-15T03:20:55.227237+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11898","last_scrub_stamp":"2024-09-15T03:20:55.227039+0000","last_deep_scrub":"37'11898","last_deep_scrub_stamp":"2024-09-15T03:20:55.227039+0000","last_clean_scrub_stamp":"2024-09-15T03:20:55.227039+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:40:04.144477+0000","scrub_duration":0.029692864999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12132","reported_seq":16355,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:59.210740+0000","last_change":"2024-09-15T03:20:59.210660+0000","last_active":"2024-09-15T03:20:59.210740+0000","last_peered":"2024-09-15T03:20:59.210740+0000","last_clean":"2024-09-15T03:20:59.210740+0000","last_became_active":"2024-09-15T03:15:01.149287+0000","last_became_peered":"2024-09-15T03:15:01.149287+0000","last_unstale":"2024-09-15T03:20:59.210740+0000","last_undegraded":"2024-09-15T03:20:59.210740+0000","last_fullsized":"2024-09-15T03:20:59.210740+0000","mapping_epoch":36,"log_start":"31'2100","ondisk_log_start":"31'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12132","last_scrub_stamp":"2024-09-15T03:20:59.210525+0000","last_deep_scrub":"37'12132","last_deep_scrub_stamp":"2024-09-15T03:20:59.210525+0000","last_clean_scrub_stamp":"2024-09-15T03:20:59.210525+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:35:55.951251+0000","scrub_duration":0.023846742000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11680","reported_seq":11810,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:47.226096+0000","last_change":"2024-09-15T03:20:47.226017+0000","last_active":"2024-09-15T03:20:47.226096+0000","last_peered":"2024-09-15T03:20:47.226096+0000","last_clean":"2024-09-15T03:20:47.226096+0000","last_became_active":"2024-09-15T03:20:20.132769+0000","last_became_peered":"2024-09-15T03:20:20.132769+0000","last_unstale":"2024-09-15T03:20:47.226096+0000","last_undegraded":"2024-09-15T03:20:47.226096+0000","last_fullsized":"2024-09-15T03:20:47.226096+0000","mapping_epoch":60,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11680","last_scrub_stamp":"2024-09-15T03:20:47.225892+0000","last_deep_scrub":"37'11680","last_deep_scrub_stamp":"2024-09-15T03:20:47.225892+0000","last_clean_scrub_stamp":"2024-09-15T03:20:47.225892+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:39:25.186283+0000","scrub_duration":0.023339927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'11881","reported_seq":12026,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:50.284524+0000","last_change":"2024-09-15T03:20:50.284424+0000","last_active":"2024-09-15T03:20:50.284524+0000","last_peered":"2024-09-15T03:20:50.284524+0000","last_clean":"2024-09-15T03:20:50.284524+0000","last_became_active":"2024-09-15T03:19:27.427201+0000","last_became_peered":"2024-09-15T03:19:27.427201+0000","last_unstale":"2024-09-15T03:20:50.284524+0000","last_undegraded":"2024-09-15T03:20:50.284524+0000","last_fullsized":"2024-09-15T03:20:50.284524+0000","mapping_epoch":52,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11881","last_scrub_stamp":"2024-09-15T03:20:50.284272+0000","last_deep_scrub":"37'11881","last_deep_scrub_stamp":"2024-09-15T03:20:50.284272+0000","last_clean_scrub_stamp":"2024-09-15T03:20:50.284272+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:00:04.015047+0000","scrub_duration":0.024328300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12007","reported_seq":12160,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T03:20:49.254331+0000","last_change":"2024-09-15T03:20:49.254253+0000","last_active":"2024-09-15T03:20:49.254331+0000","last_peered":"2024-09-15T03:20:49.254331+0000","last_clean":"2024-09-15T03:20:49.254331+0000","last_became_active":"2024-09-15T03:19:15.369090+0000","last_became_peered":"2024-09-15T03:19:15.369090+0000","last_unstale":"2024-09-15T03:20:49.254331+0000","last_undegraded":"2024-09-15T03:20:49.254331+0000","last_fullsized":"2024-09-15T03:20:49.254331+0000","mapping_epoch":50,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12007","last_scrub_stamp":"2024-09-15T03:20:49.254118+0000","last_deep_scrub":"37'12007","last_deep_scrub_stamp":"2024-09-15T03:20:49.254118+0000","last_clean_scrub_stamp":"2024-09-15T03:20:49.254118+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:13:56.947211+0000","scrub_duration":0.023478103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6969,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23908,"num_write_kb":95632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4032,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":932236,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":931707,"kb_avail":93439604,"statfs":{"total":96636764160,"available":95682154496,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":954068545},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":1.0920000000000001,"15min":1.5089999999999999},"min":{"1min":0.34799999999999998,"5min":0.191,"15min":0.16200000000000001},"max":{"1min":0.90500000000000003,"5min":34.892000000000003,"15min":34.892000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.0720000000000001,"15min":1.7589999999999999},"min":{"1min":0.36199999999999999,"5min":0.20000000000000001,"15min":0.185},"max":{"1min":0.67400000000000004,"5min":11.006,"15min":26.143000000000001},"last":0.65900000000000003}]},{"osd":1,"last update":"Sun Sep 15 03:20:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":2.7879999999999998,"15min":2.8029999999999999},"min":{"1min":0.39200000000000002,"5min":0.20300000000000001,"15min":0.19800000000000001},"max":{"1min":0.88800000000000001,"5min":27.18,"15min":40.710999999999999},"last":0.75},{"interface":"front","average":{"1min":0.66300000000000003,"5min":2,"15min":1.9099999999999999},"min":{"1min":0.40100000000000002,"5min":0.247,"15min":0.20300000000000001},"max":{"1min":0.85999999999999999,"5min":28.280999999999999,"15min":28.280999999999999},"last":0.79600000000000004}]},{"osd":2,"last update":"Sun Sep 15 03:20:57 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":1.4259999999999999,"15min":2.5950000000000002},"min":{"1min":0.46600000000000003,"5min":0.222,"15min":0.20200000000000001},"max":{"1min":0.91300000000000003,"5min":13.894,"15min":40.784999999999997},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.1259999999999999,"15min":1.7370000000000001},"min":{"1min":0.47799999999999998,"5min":0.23999999999999999,"15min":0.188},"max":{"1min":0.85699999999999998,"5min":20.236999999999998,"15min":32.377000000000002},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919304,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":918782,"kb_avail":93452536,"statfs":{"total":96636764160,"available":95695396864,"internally_reserved":0,"allocated":524288,"data_stored":228264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":940833112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.198,"15min":2.157},"min":{"1min":0.27800000000000002,"5min":0.186,"15min":0.16300000000000001},"max":{"1min":0.60199999999999998,"5min":20.701000000000001,"15min":31.175000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.81599999999999995,"15min":1.4219999999999999},"min":{"1min":0.23999999999999999,"5min":0.215,"15min":0.192},"max":{"1min":0.84799999999999998,"5min":7.75,"15min":19.329000000000001},"last":0.69099999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:20:30 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1319999999999999,"15min":1.708},"min":{"1min":0.26600000000000001,"5min":0.22700000000000001,"15min":0.19900000000000001},"max":{"1min":0.78500000000000003,"5min":10.968,"15min":32.295999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.2230000000000001,"15min":1.7829999999999999},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.20999999999999999},"max":{"1min":0.89100000000000001,"5min":11.002000000000001,"15min":32.340000000000003},"last":0.78000000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":1.0920000000000001,"15min":1.8240000000000001},"min":{"1min":0.40400000000000003,"5min":0.255,"15min":0.216},"max":{"1min":1.014,"5min":9.2240000000000002,"15min":21.600999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":1.2330000000000001,"15min":1.915},"min":{"1min":0.35199999999999998,"5min":0.27400000000000002,"15min":0.22700000000000001},"max":{"1min":0.97799999999999998,"5min":15.095000000000001,"15min":22.710000000000001},"last":0.58899999999999997}]}]},{"osd":1,"up_from":29,"seq":124554051766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926156,"kb_used_data":644,"kb_used_omap":4,"kb_used_meta":925499,"kb_avail":93445684,"statfs":{"total":96636764160,"available":95688380416,"internally_reserved":0,"allocated":659456,"data_stored":360286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":947711553},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":2.1230000000000002,"15min":2.2610000000000001},"min":{"1min":0.27700000000000002,"5min":0.183,"15min":0.16900000000000001},"max":{"1min":0.80800000000000005,"5min":28.456,"15min":28.456},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":1.286,"15min":1.8740000000000001},"min":{"1min":0.28599999999999998,"5min":0.19,"15min":0.187},"max":{"1min":0.68799999999999994,"5min":13.845000000000001,"15min":49.399999999999999},"last":0.63200000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":1.1160000000000001,"15min":1.9650000000000001},"min":{"1min":0.33800000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.80600000000000005,"5min":11.462,"15min":26.911000000000001},"last":0.872},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.6180000000000001,"15min":2.5990000000000002},"min":{"1min":0.379,"5min":0.217,"15min":0.20000000000000001},"max":{"1min":0.81799999999999995,"5min":17.332000000000001,"15min":28.145},"last":0.92900000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.524,"15min":2.4990000000000001},"min":{"1min":0.32300000000000001,"5min":0.25,"15min":0.222},"max":{"1min":0.96999999999999997,"5min":34.595999999999997,"15min":59.225999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.6499999999999999,"15min":1.8009999999999999},"min":{"1min":0.35899999999999999,"5min":0.254,"15min":0.22700000000000001},"max":{"1min":0.91100000000000003,"5min":21.401,"15min":33.444000000000003},"last":0.998}]}]},{"osd":0,"up_from":23,"seq":98784248006,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":923916,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":923387,"kb_avail":93447924,"statfs":{"total":96636764160,"available":95690674176,"internally_reserved":0,"allocated":528384,"data_stored":229214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4543,"internal_metadata":945548865},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:20:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":1.357,"15min":1.673},"min":{"1min":0.25600000000000001,"5min":0.189,"15min":0.17499999999999999},"max":{"1min":0.67000000000000004,"5min":22.384,"15min":22.384},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.0590000000000002,"15min":2.5209999999999999},"min":{"1min":0.34599999999999997,"5min":0.253,"15min":0.19900000000000001},"max":{"1min":0.746,"5min":30.433,"15min":30.433},"last":0.73999999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.6519999999999999,"15min":2.2989999999999999},"min":{"1min":0.36099999999999999,"5min":0.22700000000000001,"15min":0.16200000000000001},"max":{"1min":0.77300000000000002,"5min":30.562999999999999,"15min":30.802},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.62,"5min":1.181,"15min":1.7290000000000001},"min":{"1min":0.39500000000000002,"5min":0.23999999999999999,"15min":0.21199999999999999},"max":{"1min":0.82899999999999996,"5min":10.321999999999999,"15min":20.387},"last":0.66200000000000003}]},{"osd":3,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.878,"15min":1.3919999999999999},"min":{"1min":0.40600000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.73999999999999999,"5min":15.114000000000001,"15min":20.948},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.077,"15min":1.629},"min":{"1min":0.443,"5min":0.28899999999999998,"15min":0.22700000000000001},"max":{"1min":0.82299999999999995,"5min":15.077999999999999,"15min":30.013000000000002},"last":0.88700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":135168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:21:23.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T03:21:23.629 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T03:21:23.629 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T03:21:23.630 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T03:21:23.630 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:24.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-15T03:21:29.731 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T03:21:29.732 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T03:21:29.732 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:29.811 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T03:21:29.812 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T03:21:29.812 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:30.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~6s 2024-09-15T03:21:35.800 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-15T03:21:35.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-15T03:21:35.914 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T03:21:35.914 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T03:21:35.914 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:41.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-15T03:21:41.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-15T03:21:42.015 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T03:21:42.016 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T03:21:42.016 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T03:21:42.016 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:42.045 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T03:21:42.045 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T03:21:42.045 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:21:42.045 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:42.062 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:21:42.062 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:21:42.062 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:42.079 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:21:42.079 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:21:42.079 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:42.098 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:21:42.098 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T03:21:42.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-15T03:21:42.146 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:21:42.146 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T03:21:42.426 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:21:42.426 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T03:21:42.598 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:21:42.598 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T03:21:42.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:21:42.833 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T03:21:42.996 INFO:tasks.ceph:Archiving mon data... 2024-09-15T03:21:42.997 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213/data/mon.a.tgz 2024-09-15T03:21:42.998 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-15T03:21:43.020 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.9lqum2RRUA 2024-09-15T03:21:43.020 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.9lqum2RRUA 2024-09-15T03:21:43.117 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.9lqum2RRUA 2024-09-15T03:21:43.221 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.9lqum2RRUA is 260KB 2024-09-15T03:21:43.284 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.9lqum2RRUA 2024-09-15T03:21:43.296 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213/data/mon.b.tgz 2024-09-15T03:21:43.296 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-15T03:21:43.303 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.XKupZacFf1 2024-09-15T03:21:43.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.XKupZacFf1 2024-09-15T03:21:43.371 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.XKupZacFf1 2024-09-15T03:21:43.391 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.XKupZacFf1 is 260KB 2024-09-15T03:21:43.452 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.XKupZacFf1 2024-09-15T03:21:43.459 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213/data/mon.c.tgz 2024-09-15T03:21:43.459 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-15T03:21:43.466 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.rbFkWpdlAM 2024-09-15T03:21:43.466 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.rbFkWpdlAM 2024-09-15T03:21:43.537 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.rbFkWpdlAM 2024-09-15T03:21:43.562 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.rbFkWpdlAM is 260KB 2024-09-15T03:21:43.624 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.rbFkWpdlAM 2024-09-15T03:21:43.630 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T03:21:43.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:21:43.660 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T03:21:43.660 DEBUG:teuthology.orchestra.run.smithi060:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:21:43.689 INFO:teuthology.orchestra.run.smithi060.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:21:43.698 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T03:21:43.700 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213/remote/smithi060/crash 2024-09-15T03:21:43.701 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:21:43.717 INFO:tasks.ceph:Compressing logs... 2024-09-15T03:21:43.717 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:21:43.782 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35749.log 2024-09-15T03:21:43.782 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38529.log 2024-09-15T03:21:43.783 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42813.log 2024-09-15T03:21:43.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45379.log 2024-09-15T03:21:43.784 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35749.log.gz 2024-09-15T03:21:43.784 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38529.log: /var/log/ceph/ceph-client.admin.42813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35352.log 2024-09-15T03:21:43.784 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38529.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42813.log.gz 2024-09-15T03:21:43.784 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-15T03:21:43.785 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26070.log 2024-09-15T03:21:43.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45379.log.gz 2024-09-15T03:21:43.785 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35352.log.gz 2024-09-15T03:21:43.785 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33759.log 2024-09-15T03:21:43.785 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26340.log 2024-09-15T03:21:43.786 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33759.log: /var/log/ceph/ceph-client.admin.26070.log: gzip 0.0%-5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.33759.log.gz/var/log/ceph/ceph-client.admin.30134.log 2024-09-15T03:21:43.786 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-15T03:21:43.786 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26070.log.gz 2024-09-15T03:21:43.787 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45055.log 2024-09-15T03:21:43.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26340.log: /var/log/ceph/ceph-client.admin.30134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26340.log.gz 2024-09-15T03:21:43.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29864.log 2024-09-15T03:21:43.788 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30134.log.gz 2024-09-15T03:21:43.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34191.log 2024-09-15T03:21:43.788 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45055.log.gz 2024-09-15T03:21:43.788 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42273.log 2024-09-15T03:21:43.788 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29864.log.gz 2024-09-15T03:21:43.789 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34191.log.gz 2024-09-15T03:21:43.789 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38175.log 2024-09-15T03:21:43.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33678.log 2024-09-15T03:21:43.790 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42273.log.gz 2024-09-15T03:21:43.791 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38148.log 2024-09-15T03:21:43.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38175.log.gz 2024-09-15T03:21:43.791 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43813.log 2024-09-15T03:21:43.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33678.log.gz 2024-09-15T03:21:43.791 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28995.log 2024-09-15T03:21:43.791 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38148.log.gz 2024-09-15T03:21:43.792 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.43813.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45244.log 2024-09-15T03:21:43.792 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43813.log.gz 2024-09-15T03:21:43.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36642.log 2024-09-15T03:21:43.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.28995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28995.log.gz 2024-09-15T03:21:43.793 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45244.log.gz 2024-09-15T03:21:43.793 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27374.log 2024-09-15T03:21:43.794 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36156.log 2024-09-15T03:21:43.794 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36642.log.gz 2024-09-15T03:21:43.795 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39454.log 2024-09-15T03:21:43.795 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27374.log.gz 2024-09-15T03:21:43.795 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38202.log 2024-09-15T03:21:43.795 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36156.log.gz 2024-09-15T03:21:43.795 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29898.log 2024-09-15T03:21:43.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39454.log.gz 2024-09-15T03:21:43.796 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36750.log 2024-09-15T03:21:43.796 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38202.log.gz 2024-09-15T03:21:43.797 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44893.log 2024-09-15T03:21:43.797 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29898.log.gz 2024-09-15T03:21:43.797 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44164.log 2024-09-15T03:21:43.798 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36750.log: /var/log/ceph/ceph-client.admin.44893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36750.log.gz 2024-09-15T03:21:43.798 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T03:21:43.798 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44893.log.gz 2024-09-15T03:21:43.799 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45136.log 2024-09-15T03:21:43.799 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44164.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.44164.log.gz--verbose 2024-09-15T03:21:43.799 INFO:teuthology.orchestra.run.smithi060.stderr:-- /var/log/ceph/ceph-client.admin.31165.log 2024-09-15T03:21:43.799 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.45136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37959.log 2024-09-15T03:21:43.799 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45136.log.gz 2024-09-15T03:21:43.801 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27477.log 2024-09-15T03:21:43.801 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31165.log.gz 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43110.log 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37959.log: /var/log/ceph/ceph-client.admin.27477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27477.log.gz 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37959.log.gz 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41409.log 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39538.log 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43110.log: /var/log/ceph/ceph-client.admin.41409.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.43110.log.gz 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42327.log 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41409.log.gz 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39538.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.27587.log -- replaced with /var/log/ceph/ceph-client.admin.39538.log.gz 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42327.log.gz 2024-09-15T03:21:43.802 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27453.log 2024-09-15T03:21:43.806 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33433.log 2024-09-15T03:21:43.806 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27587.log: /var/log/ceph/ceph-client.admin.27453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27453.log.gz 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.27587.log.gz-- 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33136.log 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33433.log.gz 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22213.log 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27543.log 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33136.log.gz 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.22213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39373.log 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22213.log.gz 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44920.log 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27543.log.gz 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41111.log 2024-09-15T03:21:43.807 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39373.log.gz 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26531.log 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44920.log.gz 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42624.log 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41111.log.gz 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39484.log 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26531.log.gz 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42624.log: /var/log/ceph/ceph-client.admin.30894.log 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42624.log.gz 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39484.log.gz 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35325.log 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30894.log.gz 2024-09-15T03:21:43.808 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33813.log 2024-09-15T03:21:43.811 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34758.log 2024-09-15T03:21:43.811 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35325.log.gz 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42759.log 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33813.log.gz 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34434.log 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34758.log.gz 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28005.log 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42759.log.gz 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34434.log.gz 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33705.log 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.28005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28005.log.gz 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25865.log 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31084.log 2024-09-15T03:21:43.812 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33705.log.gz 2024-09-15T03:21:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41787.log 2024-09-15T03:21:43.813 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25865.log.gz 2024-09-15T03:21:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27521.log 2024-09-15T03:21:43.813 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31084.log.gz 2024-09-15T03:21:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-09-15T03:21:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41787.log.gz 2024-09-15T03:21:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31443.log 2024-09-15T03:21:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27521.log.gz 2024-09-15T03:21:43.814 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-09-15T03:21:43.816 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30595.log 2024-09-15T03:21:43.817 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-09-15T03:21:43.817 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31443.log.gz 2024-09-15T03:21:43.817 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2024-09-15T03:21:43.818 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30595.log.gz 2024-09-15T03:21:43.818 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36046.log 2024-09-15T03:21:43.818 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-09-15T03:21:43.818 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44839.log 2024-09-15T03:21:43.818 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2024-09-15T03:21:43.819 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.36046.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42354.log 2024-09-15T03:21:43.819 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36046.log.gz 2024-09-15T03:21:43.819 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30731.log 2024-09-15T03:21:43.820 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44839.log.gz 2024-09-15T03:21:43.820 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43002.log 2024-09-15T03:21:43.820 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42354.log.gz 2024-09-15T03:21:43.820 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.30731.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26638.log 2024-09-15T03:21:43.820 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30731.log.gz 2024-09-15T03:21:43.820 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43002.log.gz 2024-09-15T03:21:43.821 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38229.log 2024-09-15T03:21:43.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38067.log 2024-09-15T03:21:43.822 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26638.log.gz 2024-09-15T03:21:43.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36291.log 2024-09-15T03:21:43.822 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38229.log.gz 2024-09-15T03:21:43.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38067.log.gz 2024-09-15T03:21:43.822 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31355.log 2024-09-15T03:21:43.822 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45028.log 2024-09-15T03:21:43.822 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36291.log.gz 2024-09-15T03:21:43.826 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40378.log 2024-09-15T03:21:43.826 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31355.log.gz 2024-09-15T03:21:43.826 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45028.log.gz 2024-09-15T03:21:43.826 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34002.log 2024-09-15T03:21:43.826 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.40378.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30622.log 2024-09-15T03:21:43.826 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-09-15T03:21:43.826 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39022.log 2024-09-15T03:21:43.827 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34002.log.gz 2024-09-15T03:21:43.827 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41301.log 2024-09-15T03:21:43.827 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30622.log: /var/log/ceph/ceph-client.admin.39022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31725.log 2024-09-15T03:21:43.827 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39022.log.gz 2024-09-15T03:21:43.827 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30622.log.gz 2024-09-15T03:21:43.827 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41301.log.gz 2024-09-15T03:21:43.828 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27286.log 2024-09-15T03:21:43.828 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25971.log 2024-09-15T03:21:43.828 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35109.log 2024-09-15T03:21:43.829 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31725.log.gz 2024-09-15T03:21:43.829 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38968.log 2024-09-15T03:21:43.829 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27286.log.gz 2024-09-15T03:21:43.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44596.log 2024-09-15T03:21:43.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35109.log.gz 2024-09-15T03:21:43.831 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25971.log.gz 2024-09-15T03:21:43.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26846.log 2024-09-15T03:21:43.831 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38968.log.gz 2024-09-15T03:21:43.831 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25861.log 2024-09-15T03:21:43.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44596.log.gz 2024-09-15T03:21:43.831 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26846.log.gz 2024-09-15T03:21:43.832 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31057.log 2024-09-15T03:21:43.832 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34164.log 2024-09-15T03:21:43.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32165.log 2024-09-15T03:21:43.832 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25861.log.gz 2024-09-15T03:21:43.832 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31057.log.gz 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44785.log 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34164.log.gz 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45541.log 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32165.log.gz 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42030.log 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34974.log 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44785.log.gz 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45541.log.gz 2024-09-15T03:21:43.836 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28074.log 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42030.log.gz 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41490.log 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34974.log.gz 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.28074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37905.log 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28074.log.gz 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35722.log 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41490.log.gz 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26934.log 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37905.log.gz 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42867.log /var/log/ceph/ceph-client.admin.35722.log: 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35722.log.gz 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32011.log 2024-09-15T03:21:43.837 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26934.log.gz 2024-09-15T03:21:43.838 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.42867.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39565.log 2024-09-15T03:21:43.838 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42867.log.gz 2024-09-15T03:21:43.838 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32011.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.32011.log.gz--verbose 2024-09-15T03:21:43.838 INFO:teuthology.orchestra.run.smithi060.stderr:-- /var/log/ceph/ceph-client.admin.31002.log 2024-09-15T03:21:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29977.log 2024-09-15T03:21:43.853 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39565.log.gz 2024-09-15T03:21:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42219.log 2024-09-15T03:21:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31002.log.gz 2024-09-15T03:21:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45811.log 2024-09-15T03:21:43.853 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29977.log.gz 2024-09-15T03:21:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43246.log 2024-09-15T03:21:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45676.log 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42219.log.gz 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36615.log 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45811.log: /var/log/ceph/ceph-client.admin.43246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45811.log.gz 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43246.log.gz 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45676.log.gz 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22238.log 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39157.log 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36615.log: 0.0%/var/log/ceph/ceph-client.admin.22238.log: -- replaced with /var/log/ceph/ceph-client.admin.36615.log.gz 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40432.log 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22238.log.gz 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36986.log 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39157.log.gz 2024-09-15T03:21:43.854 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44812.log 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40432.log: /var/log/ceph/ceph-client.admin.36986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40432.log.gz 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36986.log.gz 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37797.log 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33624.log 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44812.log.gz 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34461.log 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37797.log.gz 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31769.log 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33624.log.gz 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25773.log 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34461.log.gz 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36399.log 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31769.log.gz/var/log/ceph/ceph-client.admin.25773.log: 2024-09-15T03:21:43.855 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25773.log.gz 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40459.log 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35541.log 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36399.log.gz 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31247.log 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40459.log.gz 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43867.log 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35541.log.gz 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42381.log 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31247.log.gz 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34029.log 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr: 27.5% -- replaced with /var/log/ceph/ceph-client.admin.43867.log.gz 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33244.log 2024-09-15T03:21:43.856 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42381.log.gz 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.34029.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26286.log 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34029.log.gz 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38583.log 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33244.log.gz 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.26286.log: -- /var/log/ceph/ceph-client.admin.43164.log 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26286.log.gz 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39400.log 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38583.log.gz 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37202.log 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43164.log.gz 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39971.log 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39400.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39400.log.gz-5 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:--verbose -- /var/log/ceph/ceph-client.admin.27132.log 2024-09-15T03:21:43.857 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37202.log.gz 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39971.log.gz 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43381.log 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27132.log.gz 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43300.log 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30512.log 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43381.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43381.log.gz 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.34731.log 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43300.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43300.log.gz 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.39890.log 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30512.log.gz 2024-09-15T03:21:43.858 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45649.log 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34731.log.gz 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39265.log 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39890.log.gz 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42408.log 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45649.log.gz 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30867.log 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39265.log.gz 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36073.log 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42408.log.gz 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37608.log 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30867.log.gz 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39727.log 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36073.log.gz 2024-09-15T03:21:43.859 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43786.log 2024-09-15T03:21:43.860 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37608.log.gz 2024-09-15T03:21:43.860 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36453.log 2024-09-15T03:21:43.860 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39727.log.gz 2024-09-15T03:21:43.860 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.43786.log: -- /var/log/ceph/ceph-client.admin.44326.log 2024-09-15T03:21:43.860 INFO:teuthology.orchestra.run.smithi060.stderr: 26.7% -- replaced with /var/log/ceph/ceph-client.admin.43786.log.gz 2024-09-15T03:21:43.860 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36453.log.gz 2024-09-15T03:21:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30975.log 2024-09-15T03:21:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44326.log.gz 2024-09-15T03:21:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37581.log 2024-09-15T03:21:43.878 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45784.log 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30975.log.gz 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45703.log 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37581.log.gz 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43732.log 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45784.log.gz 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36723.log 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45703.log.gz 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.43732.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30568.log 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43732.log.gz 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36723.log.gz 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31791.log 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30568.log.gz 2024-09-15T03:21:43.879 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42678.log 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31791.log.gz 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27763.log 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 /var/log/ceph/ceph-client.admin.42678.log: --verbose -- /var/log/ceph/ceph-client.admin.43354.log 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42678.log.gz 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27066.log 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27763.log.gz 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22280.log 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43354.log.gz 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30921.log 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27066.log.gz 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35965.log 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.22280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22280.log.gz 2024-09-15T03:21:43.880 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35028.log 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30921.log.gz 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38914.log 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35965.log.gz 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41652.log 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35028.log.gz 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33921.log 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38914.log.gz 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41652.log.gz 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36183.log 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26396.log 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33921.log.gz 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38094.log 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36183.log: /var/log/ceph/ceph-client.admin.26396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40786.log 2024-09-15T03:21:43.881 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36183.log.gz 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26396.log.gz 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31602.log 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38094.log.gz 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40187.log 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40786.log.gz 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31602.log.gz 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38860.log 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40187.log.gz 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27044.log 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38860.log.gz 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30055.log 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27044.log.gz 2024-09-15T03:21:43.882 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38698.log 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30055.log.gz 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32099.log 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33379.log 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38698.log.gz 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32099.log.gz 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39754.log 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33298.log 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33379.log.gz 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44542.log 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39754.log.gz 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22101.log 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33298.log.gz 2024-09-15T03:21:43.883 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44542.log.gz 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37473.log 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.22101.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.22101.log.gz--verbose 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:-- /var/log/ceph/ceph-client.admin.45298.log 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38310.log 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40296.log 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37473.log.gz 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45298.log.gz 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41436.log 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38310.log.gz 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26178.log 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40296.log.gz 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41436.log.gz 2024-09-15T03:21:43.884 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30537.log 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27983.log 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26178.log.gz 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37311.log 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30537.log: /var/log/ceph/ceph-client.admin.27983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38725.log 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27983.log.gz 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31901.log 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37311.log.gz 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38725.log.gz 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43273.log 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32253.log 2024-09-15T03:21:43.885 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31901.log.gz 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31703.log 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43273.log.gz 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32253.log.gz 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37770.log 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36858.log 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31703.log: /var/log/ceph/ceph-client.admin.37770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31703.log.gz 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37770.log.gz-5 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:--verbose -- /var/log/ceph/ceph-client.admin.34893.log 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33487.log 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36858.log.gz 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31442.log 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34893.log.gz 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33487.log.gz 2024-09-15T03:21:43.886 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40350.log 2024-09-15T03:21:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.31442.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35406.log 2024-09-15T03:21:43.887 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31442.log.gz 2024-09-15T03:21:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45109.log 2024-09-15T03:21:43.887 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40350.log.gz 2024-09-15T03:21:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36885.log 2024-09-15T03:21:43.887 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35406.log.gz 2024-09-15T03:21:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45109.log.gz 2024-09-15T03:21:43.887 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33352.log 2024-09-15T03:21:43.905 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45514.log 2024-09-15T03:21:43.905 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36885.log.gz 2024-09-15T03:21:43.905 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33352.log.gz 2024-09-15T03:21:43.905 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42165.log 2024-09-15T03:21:43.905 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44758.log 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45514.log.gz 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31274.log 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42165.log.gz 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44758.log.gz 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37284.log 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36480.log 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31274.log.gz 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37284.log.gz 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41274.log 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31652.log 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36480.log.gz 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44110.log 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41274.log.gz 2024-09-15T03:21:43.906 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34326.log 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31652.log.gz 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 /var/log/ceph/ceph-client.admin.44110.log: --verbose -- /var/log/ceph/ceph-client.admin.36318.log 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44110.log.gz 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26743.log 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34326.log.gz 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36318.log.gz 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26259.log 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37662.log 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26743.log.gz 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37365.log /var/log/ceph/ceph-client.admin.26259.log: 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26259.log.gz 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2024-09-15T03:21:43.907 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37662.log.gz 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45406.log 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37365.log.gz 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42246.log 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43570.log 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45406.log.gz 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42246.log.gz 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42597.log 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25962.log 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43570.log: 0.0%/var/log/ceph/ceph-client.admin.42597.log: -- replaced with /var/log/ceph/ceph-client.admin.43570.log.gz 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43948.log 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42597.log.gz 2024-09-15T03:21:43.908 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25962.log.gz 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34839.log 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39917.log 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43948.log.gz 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34839.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34839.log.gz 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.42786.log 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39917.log: /var/log/ceph/ceph-client.admin.42786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39917.log.gz 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32231.log 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42786.log.gz 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30812.log 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31328.log 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32231.log.gz 2024-09-15T03:21:43.909 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36019.log 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30812.log.gz 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31328.log.gz 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42138.log 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35857.log 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36019.log.gz 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39238.log 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42138.log: 0.0%/var/log/ceph/ceph-client.admin.35857.log: -- replaced with /var/log/ceph/ceph-client.admin.42138.log.gz 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35857.log.gz 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37878.log 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39238.log.gz 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39700.log 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37878.log.gz 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36588.log 2024-09-15T03:21:43.910 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39700.log.gz 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44704.log 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36588.log.gz 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41192.log 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44704.log.gz 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26078.log 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41192.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.27895.log -- replaced with /var/log/ceph/ceph-client.admin.41192.log.gz 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37419.log 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26078.log.gz 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31599.log 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27895.log: /var/log/ceph/ceph-client.admin.37419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34110.log 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27895.log.gz 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37419.log.gz 2024-09-15T03:21:43.911 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31599.log.gz 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34110.log.gz 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34083.log 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45757.log 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34083.log.gz 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37040.log 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45757.log.gz 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31525.log 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37040.log: /var/log/ceph/ceph.audit.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37040.log.gz 2024-09-15T03:21:43.912 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42435.log 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31525.log.gz 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38475.log 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42435.log.gz 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45217.log 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38475.log.gz 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26103.log 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36534.log 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45217.log.gz 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26103.log.gz 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26978.log 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32077.log 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36534.log: /var/log/ceph/ceph-client.admin.26978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26477.log 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36534.log.gz 0.0% 2024-09-15T03:21:43.913 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26978.log.gz 2024-09-15T03:21:43.914 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32077.log.gz 2024-09-15T03:21:43.914 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22162.log 2024-09-15T03:21:43.914 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26477.log.gz 2024-09-15T03:21:43.914 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38752.log 2024-09-15T03:21:43.914 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.22162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22162.log.gz 2024-09-15T03:21:43.932 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34137.log 2024-09-15T03:21:43.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38752.log.gz 2024-09-15T03:21:43.932 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31138.log 2024-09-15T03:21:43.932 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35911.log 2024-09-15T03:21:43.932 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34137.log.gz 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31138.log.gz 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43975.log 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35911.log: /var/log/ceph/ceph-client.admin.43975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26528.log 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35911.log.gz 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43975.log.gz 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27264.log 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26528.log.gz 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30161.log 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27264.log.gz 2024-09-15T03:21:43.933 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36831.log 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30057.log 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30161.log.gz 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37257.log 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36831.log.gz 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30057.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.30057.log.gz/var/log/ceph/ceph-client.admin.27308.log 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28027.log 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37257.log.gz 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41328.log 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27308.log: 0.0%/var/log/ceph/ceph-client.admin.28027.log: -- replaced with /var/log/ceph/ceph-client.admin.27308.log.gz 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28027.log.gz 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44245.log 2024-09-15T03:21:43.934 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41328.log.gz 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34407.log 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44245.log.gz 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37067.log 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34407.log.gz 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33867.log 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37067.log.gz 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33867.log.gz 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30758.log 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30438.log 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37500.log 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30758.log.gz 2024-09-15T03:21:43.935 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30438.log.gz 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34353.log 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37500.log.gz 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34623.log 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34353.log.gz 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45163.log 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34623.log.gz 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27330.log 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45163.log.gz 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26448.log 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27330.log.gz 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37148.log 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26448.log.gz 2024-09-15T03:21:43.936 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38833.log 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30431.log 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37148.log: /var/log/ceph/ceph-client.admin.38833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40513.log 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38833.log.gz 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37148.log.gz 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30431.log.gz 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39319.log 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40513.log.gz 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32055.log 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33190.log 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32055.log.gz 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40895.log 2024-09-15T03:21:43.937 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33190.log.gz 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26802.log 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40895.log.gz 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25800.log 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37689.log 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26802.log.gz 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31989.log 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25800.log.gz 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37446.log 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37689.log.gz 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31989.log.gz 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27697.log 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37446.log.gz 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38040.log 2024-09-15T03:21:43.938 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27697.log.gz 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30704.log 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38040.log.gz 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35992.log 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30704.log.gz 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39673.log 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35992.log.gz 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38502.log 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39673.log.gz 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35695.log 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38502.log: /var/log/ceph/ceph-client.admin.35695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38502.log.gz 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35695.log.gz 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35568.log 2024-09-15T03:21:43.939 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39781.log 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40486.log: /var/log/ceph/ceph-client.admin.35568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35568.log.gz 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38121.log 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39781.log.gz 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32033.log 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38121.log.gz 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43029.log 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32033.log.gz 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38995.log 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34704.log 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43029.log.gz 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34380.log 2024-09-15T03:21:43.940 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38995.log: /var/log/ceph/ceph-client.admin.34704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38995.log.gz 2024-09-15T03:21:43.941 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34704.log.gz 2024-09-15T03:21:43.941 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2024-09-15T03:21:43.941 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34380.log.gz 2024-09-15T03:21:43.961 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42489.log 2024-09-15T03:21:43.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38644.log 2024-09-15T03:21:43.961 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2024-09-15T03:21:43.961 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40732.log 2024-09-15T03:21:43.961 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42489.log.gz 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33516.log 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38644.log.gz 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30277.log 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40732.log.gz 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.33516.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30357.log 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33516.log.gz 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30277.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30277.log.gz 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.27565.log 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27609.log 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30357.log.gz 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27565.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27565.log.gz 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33786.log 2024-09-15T03:21:43.962 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27609.log.gz 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31857.log 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33786.log.gz 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26778.log 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43218.log 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31857.log.gz 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37338.log 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26778.log.gz 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41138.log 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43218.log.gz 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39427.log 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37338.log.gz 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41138.log.gz 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38779.log 2024-09-15T03:21:43.963 INFO:teuthology.orchestra.run.smithi060.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.admin.30537.log.gz 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27426.log 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39427.log.gz 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26232.log 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38779.log.gz 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27426.log.gz 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25745.log 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35163.log 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26232.log.gz 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34650.log 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25745.log.gz 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27939.log 2024-09-15T03:21:43.964 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35163.log.gz 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27675.log 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34650.log.gz 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41625.log 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27939.log.gz 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.27675.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31747.log 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27675.log.gz 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37824.log 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41625.log.gz 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42975.log 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31747.log.gz 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37824.log.gz 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42975.log.gz 2024-09-15T03:21:43.965 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38364.log 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41057.log 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38364.log.gz 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40241.log 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39836.log 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41057.log.gz 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30268.log 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40241.log.gz 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36939.log 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39836.log.gz 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31111.log 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30268.log: /var/log/ceph/ceph-client.admin.36939.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30268.log.gz 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.40323.log 2024-09-15T03:21:43.966 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36939.log.gz 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42705.log 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31111.log.gz 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40323.log: /var/log/ceph/ceph-client.admin.36210.log 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40323.log.gz 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42705.log.gz 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44677.log 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32275.log 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36210.log: /var/log/ceph/ceph-client.admin.44677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44677.log.gz 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36210.log.gz 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35217.log 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41868.log 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32275.log.gz 2024-09-15T03:21:43.967 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29899.log 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35217.log.gz 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45622.log 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41868.log: /var/log/ceph/ceph-client.admin.29899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29899.log.gz 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36912.log 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45622.log: 95.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.45622.log.gz-5 --verbose 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:-- /var/log/ceph/ceph-client.admin.44083.log 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36912.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35244.log 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36912.log.gz 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44083.log.gz 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44299.log 2024-09-15T03:21:43.968 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27242.log 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35244.log.gz 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45730.log 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44299.log.gz 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30196.log 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27242.log.gz 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45730.log.gz 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30649.log 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30061.log 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30196.log.gz 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31219.log 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30649.log.gz 2024-09-15T03:21:43.969 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31192.log 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30061.log: /var/log/ceph/ceph-client.admin.31219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30061.log.gz 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31219.log.gz 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45595.log 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42003.log 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31192.log.gz 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34596.log 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44488.log 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45595.log.gz 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42003.log.gz 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41706.log 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34596.log.gz 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42300.log 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44488.log.gz 2024-09-15T03:21:43.970 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz 2024-09-15T03:21:43.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33460.log 2024-09-15T03:21:43.971 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-09-15T03:21:43.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42300.log.gz 2024-09-15T03:21:43.971 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33460.log.gz 2024-09-15T03:21:43.986 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26476.log 2024-09-15T03:21:43.986 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-09-15T03:21:43.986 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41949.log 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31945.log 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26476.log.gz 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41949.log.gz 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30350.log 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31945.log.gz 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39809.log 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27499.log 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30350.log.gz 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25975.log 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39809.log.gz 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36264.log 2024-09-15T03:21:43.987 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27499.log.gz 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36345.log 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25975.log.gz 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40621.log 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36264.log.gz 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41976.log 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36345.log.gz 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33406.log 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40621.log.gz 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44407.log 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33406.log.gz 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30195.log 2024-09-15T03:21:43.988 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44407.log.gz 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30195.log.gz 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35649.log 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33271.log 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35649.log.gz 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26077.log 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45082.log 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33271.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.33271.log.gz--verbose 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:-- /var/log/ceph/ceph-client.admin.35803.log 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26077.log.gz 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43651.log 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45082.log.gz 2024-09-15T03:21:43.989 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31301.log 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35803.log.gz 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.43651.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.22188.log 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43651.log.gz 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36372.log 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31301.log.gz 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38391.log /var/log/ceph/ceph-client.admin.22188.log: 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22188.log.gz 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45863.log 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36372.log.gz 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38391.log.gz 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30274.log 2024-09-15T03:21:43.990 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17163.log 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45863.log.gz 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30274.log.gz 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33651.log 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29976.log 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.tmp-client.admin.17163.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17163.log.gz 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39346.log 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33651.log.gz 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26868.log 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29976.log.gz 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31879.log 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39346.log.gz 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35460.log 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26868.log.gz 2024-09-15T03:21:43.991 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31879.log.gz 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.35460.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42111.log 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35460.log.gz 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41841.log 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42111.log.gz 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28968.log 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41571.log 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41841.log.gz 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.28968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-09-15T03:21:43.992 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28968.log.gz 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41571.log.gz 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35830.log 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31813.log 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45271.log 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35830.log: gzip /var/log/ceph/ceph-client.admin.31813.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31523.log 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35830.log.gz 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31813.log.gz 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33217.log 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45271.log.gz 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31523.log.gz 2024-09-15T03:21:43.993 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27110.log 2024-09-15T03:21:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41517.log 2024-09-15T03:21:43.994 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33217.log.gz 2024-09-15T03:21:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-09-15T03:21:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27110.log.gz 2024-09-15T03:21:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32121.log 2024-09-15T03:21:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41517.log.gz 2024-09-15T03:21:43.994 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40269.log 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35271.log 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32121.log.gz 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33163.log 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40269.log.gz 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.35271.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26205.log 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35271.log.gz 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.33163.log: -- /var/log/ceph/ceph-client.admin.43191.log 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33163.log.gz 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45487.log 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26205.log.gz 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37851.log 2024-09-15T03:21:44.013 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43191.log.gz 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37094.log 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45487.log.gz 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37851.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.35379.log 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37851.log.gz 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37094.log.gz 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34056.log 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33948.log 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35379.log.gz 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44218.log 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34056.log.gz 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41003.log 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33948.log.gz 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44218.log.gz 2024-09-15T03:21:44.014 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34920.log 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41003.log.gz 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41219.log 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34920.log.gz 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39998.log 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41219.log.gz 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30439.log 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39998.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39998.log.gz 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.26585.log 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43678.log 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30439.log.gz 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 /var/log/ceph/ceph-client.admin.26585.log: --verbose -- /var/log/ceph/ceph-client.admin.44434.log 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26585.log.gz 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44056.log 2024-09-15T03:21:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43678.log: /var/log/ceph/ceph-client.admin.44434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43678.log.gz 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44434.log.gz 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37743.log 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25827.log 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44056.log.gz 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37013.log 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37743.log.gz 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25827.log.gz 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43840.log 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37013.log: /var/log/ceph/ceph-client.admin.43840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37013.log.gz 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43840.log.gz 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33597.log 2024-09-15T03:21:44.016 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38556.log 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37175.log 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33597.log.gz 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43056.log 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38556.log.gz 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37175.log.gz 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41084.log 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43056.log.gz 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42651.log 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45568.log 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41084.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41084.log.gz-5 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:--verbose -- /var/log/ceph/ceph-client.admin.45433.log 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42651.log: /var/log/ceph/ceph-client.admin.45568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42651.log.gz 2024-09-15T03:21:44.017 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45568.log.gz 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33543.log 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25866.log 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45433.log.gz 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29980.log 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33543.log.gz 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44191.log 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25866.log.gz 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43489.log 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29980.log.gz 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44191.log.gz 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34218.log 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43489.log.gz 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34542.log 2024-09-15T03:21:44.018 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34218.log.gz 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43624.log 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34542.log.gz 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34299.log 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35884.log 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43624.log.gz 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27154.log 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34299.log: /var/log/ceph/ceph-client.admin.35884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38887.log 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34299.log.gz 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35884.log.gz 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43083.log 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27154.log.gz 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33840.log 2024-09-15T03:21:44.019 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38887.log.gz 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43083.log.gz 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44353.log 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37121.log 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33840.log.gz 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44137.log 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44353.log.gz/var/log/ceph/ceph-client.admin.37121.log: 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37121.log.gz 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31444.log 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38671.log 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44137.log.gz 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26639.log 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31444.log.gz 2024-09-15T03:21:44.020 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34272.log 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38671.log: /var/log/ceph/ceph-client.admin.26639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36127.log 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38671.log.gz 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26639.log.gz 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40133.log 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34272.log.gz 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44623.log 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35001.log 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36127.log.gz 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40133.log.gz 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44623.log.gz 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31409.log 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27198.log 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35001.log.gz 2024-09-15T03:21:44.021 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31409.log.gz 2024-09-15T03:21:44.022 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44569.log 2024-09-15T03:21:44.044 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34947.log 2024-09-15T03:21:44.044 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27198.log.gz 2024-09-15T03:21:44.044 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27961.log 2024-09-15T03:21:44.044 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44569.log.gz 2024-09-15T03:21:44.044 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36100.log 2024-09-15T03:21:44.044 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34947.log.gz 2024-09-15T03:21:44.044 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35082.log 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27961.log.gz 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26689.log 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36100.log.gz 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44461.log 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35082.log.gz 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36426.log 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26689.log.gz 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40813.log 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44461.log.gz 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41679.log 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36426.log.gz 2024-09-15T03:21:44.045 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43759.log 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40813.log.gz 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44947.log 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41679.log.gz 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43759.log.gz 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40648.log 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33570.log 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44947.log.gz 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27873.log 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40648.log.gz 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2024-09-15T03:21:44.046 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33570.log.gz 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35298.log 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27873.log.gz 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35776.log 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39049.log 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35298.log.gz 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.35776.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41760.log 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35776.log.gz 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43543.log 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39049.log.gz 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36669.log 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41760.log.gz 2024-09-15T03:21:44.047 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45838.log 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43543.log.gz 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40405.log 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36669.log.gz 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27631.log 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45838.log.gz 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44515.log 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gz 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27741.log 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27631.log.gz 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44515.log.gz 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25858.log 2024-09-15T03:21:44.048 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27741.log.gz 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34515.log 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42516.log 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25858.log.gz 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34515.log.gz 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39103.log 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41895.log 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42516.log.gz 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.39103.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33975.log 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39103.log.gz 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.41895.log: -- /var/log/ceph/ceph-client.admin.42840.log 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41895.log.gz 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35514.log 2024-09-15T03:21:44.049 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33975.log.gz 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42840.log.gz 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26956.log 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35514.log.gz 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30358.log 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26956.log.gz 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43894.log 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30358.log.gz 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44002.log 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr: 95.5%gzip -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:-5 /var/log/ceph/ceph-client.admin.43894.log: --verbose -- /var/log/ceph/ceph-client.admin.38256.log 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43894.log.gz 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35136.log 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44002.log.gz 2024-09-15T03:21:44.050 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34866.log 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38256.log.gz 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.35136.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41463.log 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35136.log.gz 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34866.log.gz 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41463.log.gz 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40106.log 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31029.log 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40106.log.gz 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34488.log 2024-09-15T03:21:44.051 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31835.log 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31029.log.gz 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34488.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34488.log.gz 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.32209.log 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40922.log 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31835.log.gz 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32209.log.gz 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40052.log 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30785.log 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40922.log.gz 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41165.log 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40052.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40052.log.gz 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35487.log 2024-09-15T03:21:44.052 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30785.log.gz 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43408.log 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41165.log.gz 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29893.log 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35487.log.gz 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38941.log 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43408.log: /var/log/ceph/ceph-client.admin.29893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31967.log 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43408.log.gz 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29893.log.gz 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42570.log 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38941.log.gz 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31967.log.gz 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31923.log 2024-09-15T03:21:44.053 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42570.log.gz 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37554.log 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31923.log.gz 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36777.log 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37554.log.gz 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39619.log 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27829.log 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39619.log.gz 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36777.log.gz 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27829.log.gz 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44974.log 2024-09-15T03:21:44.054 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2024-09-15T03:21:44.068 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32143.log 2024-09-15T03:21:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44974.log.gz 2024-09-15T03:21:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34245.log 2024-09-15T03:21:44.069 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42084.log 2024-09-15T03:21:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32143.log: /var/log/ceph/ceph-client.admin.34245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32143.log.gz 2024-09-15T03:21:44.070 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34245.log.gz 2024-09-15T03:21:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45190.log 2024-09-15T03:21:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30948.log 2024-09-15T03:21:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32187.log 2024-09-15T03:21:44.070 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42084.log.gz 2024-09-15T03:21:44.070 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45190.log: /var/log/ceph/ceph-client.admin.30948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37932.log 2024-09-15T03:21:44.070 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45190.log.gz 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30948.log.gz 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.32187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32187.log.gz 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26912.log 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37932.log.gz 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33732.log 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26716.log 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26912.log.gz 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33732.log.gz 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36507.log 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30192.log 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26716.log: /var/log/ceph/ceph-client.admin.36507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26716.log.gz 2024-09-15T03:21:44.071 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42543.log 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36507.log.gz 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36804.log 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30192.log.gz 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27176.log 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42543.log.gz 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41922.log 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36804.log.gz 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36237.log 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27176.log.gz 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41922.log.gz 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35622.log 2024-09-15T03:21:44.072 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36237.log.gz 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41733.log 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27653.log 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41733.log.gz 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38283.log 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35622.log.gz 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40540.log 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27653.log: /var/log/ceph/ceph-client.admin.38283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27653.log.gz 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38421.log 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38283.log.gz 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40540.log.gz 2024-09-15T03:21:44.073 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38337.log 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31521.log 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38421.log.gz 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38337.log.gz 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45460.log 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31521.log.gz 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44650.log 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27399.log 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45460.log.gz 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35595.log 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44650.log.gz 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33894.log 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27399.log.gz 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31679.log 2024-09-15T03:21:44.074 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35595.log.gz 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33894.log.gz 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26367.log 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.31679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31679.log.gz 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43921.log 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30676.log 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26367.log.gz 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.43921.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38610.log 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43921.log.gz 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28101.log 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30676.log.gz 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38610.log.gz 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38806.log 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.28101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34569.log 2024-09-15T03:21:44.075 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28101.log.gz 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38806.log.gz 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37392.log 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34569.log.gz 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44380.log 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21956.log 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37392.log.gz 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44380.log.gz 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27088.log 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.21956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43435.log 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip /var/log/ceph/ceph-client.admin.27088.log: -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27088.log.gz 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37635.log 2024-09-15T03:21:44.076 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43435.log.gz 2024-09-15T03:21:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44731.log 2024-09-15T03:21:44.077 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.37635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37635.log.gz 2024-09-15T03:21:44.090 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42894.log 2024-09-15T03:21:44.090 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30840.log 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44731.log.gz 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26424.log 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42894.log.gz 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.30840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27785.log 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30840.log.gz 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45325.log 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26424.log.gz 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27785.log.gz 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43327.log 2024-09-15T03:21:44.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45325.log.gz 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35055.log 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43327.log.gz 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36696.log 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35055.log.gz 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.36696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27851.log 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36696.log.gz 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25976.log 2024-09-15T03:21:44.092 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40160.log /var/log/ceph/ceph-client.admin.27851.log: 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27851.log.gz 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44272.log 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.25976.log: /var/log/ceph/ceph-client.admin.40160.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40160.log.gz 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25976.log.gz 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35433.log 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37527.log 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44272.log.gz 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr: 26.4% -- replaced with /var/log/ceph/ceph-client.admin.21956.log.gz 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45352.log 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43462.log 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35433.log.gz 2024-09-15T03:21:44.093 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37527.log.gz 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27719.log 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45352.log.gz 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34812.log 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43462.log.gz 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33325.log 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27719.log.gz 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34812.log.gz 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42921.log 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40214.log 2024-09-15T03:21:44.094 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.33325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33325.log.gz 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42921.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42921.log.gz 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40759.log 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43516.log 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40214.log.gz 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39863.log 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40759.log: /var/log/ceph/ceph-client.admin.43516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43516.log.gz 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27000.log 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2024-09-15T03:21:44.095 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39863.log.gz 2024-09-15T03:21:44.096 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26890.log 2024-09-15T03:21:44.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27000.log.gz 2024-09-15T03:21:44.096 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22384.log 2024-09-15T03:21:44.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.45001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz 2024-09-15T03:21:44.096 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35938.log 2024-09-15T03:21:44.096 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26890.log: /var/log/ceph/ceph-client.admin.22384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22384.log.gz 2024-09-15T03:21:44.096 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26890.log.gz 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27807.log 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35938.log.gz 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26313.log 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27807.log.gz 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43597.log 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26313.log.gz 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37986.log 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43597.log.gz 2024-09-15T03:21:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43705.log 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37986.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37986.log.gz 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27022.log 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42948.log 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43705.log: /var/log/ceph/ceph-client.admin.27022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27022.log.gz 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43705.log.gz 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26824.log 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27352.log 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42948.log.gz 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26824.log.gz 2024-09-15T03:21:44.098 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27917.log 2024-09-15T03:21:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27352.log.gz 2024-09-15T03:21:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39944.log 2024-09-15T03:21:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39646.log 2024-09-15T03:21:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27917.log: /var/log/ceph/ceph-client.admin.39944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39511.log 2024-09-15T03:21:44.099 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27917.log.gz 2024-09-15T03:21:44.099 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39944.log.gz 2024-09-15T03:21:44.099 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gz 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26584.log 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38448.log 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39511.log.gz 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.26584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26584.log.gz 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42732.log 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31382.log 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38448.log.gz 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42192.log 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42732.log: /var/log/ceph/ceph-client.admin.31382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42732.log.gz 2024-09-15T03:21:44.107 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28049.log 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31382.log.gz 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42192.log.gz 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.28049.log: /var/log/ceph/ceph-client.admin.39184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27220.log 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28049.log.gz 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36561.log 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37229.log 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.27220.log: /var/log/ceph/ceph-client.admin.36561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42057.log 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27220.log.gz 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36561.log.gz 2024-09-15T03:21:44.108 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43137.log 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37229.log.gz 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42057.log.gz 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38013.log 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.43137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34677.log 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43137.log.gz 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38013.log.gz 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44866.log 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39130.log 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34677.log.gz 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34785.log 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.44866.log: /var/log/ceph/ceph-client.admin.39130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44866.log.gz 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39130.log.gz 2024-09-15T03:21:44.109 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34785.log.gz 2024-09-15T03:21:44.736 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T03:21:45.054 INFO:teuthology.orchestra.run.smithi060.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T03:21:45.747 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T03:21:46.029 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T03:21:46.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-15T03:29:26.962 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T03:31:45.487 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T03:32:06.082 INFO:teuthology.orchestra.run.smithi060.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T03:32:38.244 INFO:teuthology.orchestra.run.smithi060.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T03:32:38.245 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-15T03:32:38.245 INFO:teuthology.orchestra.run.smithi060.stderr:real 10m54.481s 2024-09-15T03:32:38.245 INFO:teuthology.orchestra.run.smithi060.stderr:user 9m44.179s 2024-09-15T03:32:38.246 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m26.117s 2024-09-15T03:32:38.246 INFO:tasks.ceph:Archiving logs... 2024-09-15T03:32:38.247 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213/remote/smithi060/log 2024-09-15T03:32:38.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:33:37.850 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T03:33:37.861 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T03:33:37.861 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:33:37.923 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T03:33:37.932 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T03:33:37.932 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:33:38.014 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T03:33:38.015 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2024-09-15T03:33:38.015 INFO:teuthology.orchestra.run.smithi060.stdout:*hv01.front.sepi 67.205.162.81 3 u 33 64 377 0.091 4.314 1.994 2024-09-15T03:33:38.015 INFO:teuthology.orchestra.run.smithi060.stdout:+hv02.front.sepi 63.231.80.2 3 u 30 64 377 0.100 3.874 2.679 2024-09-15T03:33:38.015 INFO:teuthology.orchestra.run.smithi060.stdout:+hv03.front.sepi 74.6.168.72 3 u 50 64 377 0.091 4.696 1.972 2024-09-15T03:33:38.015 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-15T03:33:38.016 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T03:33:38.024 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T03:33:38.025 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T03:33:38.034 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T03:33:38.049 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T03:33:38.058 INFO:teuthology.task.internal:Duration was 2231.756535 seconds 2024-09-15T03:33:38.058 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T03:33:38.067 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T03:33:38.067 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:33:38.220 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T03:33:38.220 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-09-15T03:33:38.220 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:33:38.246 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T03:33:38.246 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:33:38.286 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T03:33:38.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:33:38.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T03:33:38.574 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T03:33:38.574 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:33:38.625 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T03:33:38.636 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:33:38.655 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-09-15T03:33:38.703 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:33:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:33:38.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T03:33:38.718 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T03:33:38.719 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905213/remote/smithi060 2024-09-15T03:33:38.719 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:33:38.776 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T03:33:38.776 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:33:38.782 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T03:33:38.792 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T03:33:38.792 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T03:33:38.801 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T03:33:38.801 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:33:38.809 INFO:teuthology.orchestra.run.smithi060.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:33 /home/ubuntu/cephtest 2024-09-15T03:33:38.811 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T03:33:38.821 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T03:33:38.841 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} duration: 2231.756535053253 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T03:33:38.842 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:33:38.926 INFO:teuthology.run:pass